I have a huge amount of saved HTML pages in my PC. I had parsed the the HTML page and got the image src. I need to store the images in every HTML page in a specific structure in separate directory. I tried out NET::HTTP.get but i am getting a error of Filename too long. Is there any way to do this ??
Below are the ways i tried out.
Method 1:
{
require 'open-uri'
def save_image(imgsrc)
File.open("images/img1","w") do |f|
asdf = open(imgsrc).read
f.write(asdf)
end
end
}
Method 2:
{
require 'NET::HTTP'
def save_image(imgsrc)
File.open("images/img1","w") do |f|
asdf = Net::HTTP.get_response(URI.parse(imgsrc)
f.write(asdf)
end
end
}
imgsrc => data:image/jpeg;base64,/9j/4AAQSkZJRgABAQEAYABgAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5PTgyPC4zNDL/2wBDAQkJCQwLDBgNDRgyIRwhMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjL/wAARCABxAHEDASIAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwD3+iiigAooooAKKKKACiiigAoNFYnibWG0a2sJF6XF/BbsfRWbB/SgDbooooAKKKKACiiigApDRmqGs6vbaJpkl/d7/KjIGEXLEk4AA7nJoA0KKgtLqG9tUuIHDxOMqw71NQAtFGaTPFIBaSjNGaYC1ynjyxOoWOkwjPOqW5JHb5s5/TH411WaY4R1G8AgEEZ7HtSEySmeYvmiPcN5G7HfFKTyB3rnDcyt8RhCpHkR6Yd4z0YyD+mPzpjOkozTd6lymRuAyRnmmieNpmhEiGRRkoGGQPcUrgS0UUUwOa8XeM7DwfFaPepJI1zJtVY+oUfeb6DI/OpfEDxz/wBh8h4ptQjPqGGx2H6gVi+KLGDVvEt1azxrIE0SV03DO1vMBBH5VVW8x4F8FXMr/vFubJTk8njY36EmpZLZ1ukmHTdAYudkVs028+gV2yf0JrK8EeNE8XxXpNq1rLbyDEbHJMbZ2t+ODVvxSRZeDdV2twY5Cc/7bc/+hGsrRbZbH4gPFGgjE2kI7ADGdjIo/maYXLvjrxPP4Y0eOeytftN5NJsji7YA3MT7AD9ap694svD4IsdT0SASX2oBRBG3Ow7S759cBWq/rca3Xi7RbZxuUwXJZcfwlQp/mPzrl/DM6pb+EopTgfbdQiAPTIMmB+WRSYX1L3iPxHro0HRf7ISNdRuYRdXJPKpGoXf/AOPOPwzTvH2ta4ltZWvh0hbtomvZ3P8ADCmMj8S36VJFa+TrtlpUzYlk0SeFQe53pn9Oa0LCAXHii9imGWg0yC3Yf7xYn/PtQGpX1bxFdXvw9h1DSUP9o6jGkNsn92V+D/3z8x/4DXHL42v18I+G9Put51G61EWtxKeyxypn8SGX9a1vCrEweHbBzkQapeEZ9kkZf0ko8SaKtpottNJAFk/4SNJo+OVUyhf1Cg0hNuxV8UarrcHxBt9Vtpyujabcw2U8QP3zIMuSPYH9Kzb6TxBefHRre1WSO0jlhebb0MI2Ek+xK1ueKLK4t/Cnim4niMYfUDLFn+JQFG7+f5Vv24A+J943/UMXP/fa0w1uc/ps2oR/EWPXHumbT9TubjT1g7IIuEYfVkb86n8MWE3/AAm8viGS6kkGqm6jWIn5USKTav6Kv61Sj1C2Tw34SAmT7Q+srhARu+aZ8/zra+Htzb6npCpuxc6XeXMTr7M7EfoR+VCBHdZoooqizE1xLDT7bUdYnKxytaeQ0jH+EbiB+bfyrz+6iefwDpflj97Y6Y1xGP8Ab8xAv6Bvzrf8bxf21r+maHISbSOCbUbhM/f8sYQH23HNUNAH2iysLLaGaXQJGVfUrKP6kVD3Ie50NtHdalr9wLq5S50W7skmitmQYUnb379CfxqrY6lbah8U51tpRJ9l06SCUjoHEkZI/DcKzvDWsND4UuLrful0zTfIyf70byqv57F/Oo/BOlppXiTTIxzLPobXE7nq8jyozEnuecfhTQHcaxf2Wj2M2q3pVI7eM5kI5AOOB9SB+leP6vJcw/D/AEnV7IlZrHVLm6T/AL/Nx+RJPsDXoXjKEalrHhvRpADb3F400ynoyxIXAPsTisXTbJNR8AyWwUNPHdXk6Rf31WV1dfxVyv8AwKh6g9WZHj3WpoJ/C/jCxB2xxrKVHeN8bx/48o/4FXa6NMs3j3X3Vso1nZsv0Ikrkxp0ep/C20gVt8tjauHH96Eu6H8vLDfVBWz4LkZvEczN/wAttFsH+pCtn/0IUkC3Mrw0SniXT4Sf9Xqd2B/4DJ/ga6b4gf8AIFsf+wpaf+jRXK6U5g8V6VMfuTa3eJ/5AKj9RXVfEA40Wx/7Cdp/6MFPoC2MD4qa8yWkmhW0XmM1ubi7ftFGOF/Etj8q2i2z4h6iw6jSc/8AjwrlfEiCe28f3zcyBre2X2VBnH65rqtu/wCIuoJ/e0nH/jy0kPqcx4K+GukXMGn+Ip5bprtLlp1TzPkVllOOP+AiovBsc+ifFnUbME/ZdQjl+X0kib+ePm+jiuw+HVysvgq3yQDHcTxn6+c+P5is5bdZvGvhrW1UK2oQu8iDoHEByfxGwf8AABTQktmd/miiiqLOH1iVIviBcO5AC+H5zknpiRCaxtCmFlf+C7hzhLm0lsyfc5YD80FT+OvD1/rfjfRktfOjt5YWiuZUyB5OcupPuMDHfNZlz4P1u1+Hdt5Ku+r2V891AinJQb+MfgAfxNTrcze5IYE0638daYM5KRzRjP8AAcr/ADGfxrc0W+gn8d6aiOv/ACABgZ9WjYD8qxZvBWtPdWt9PI0lxNZTyX+xuJJTysf0yVA/3TUngv4fX2lX3h/VLqTEkMcz3Cs3zAsoVF+gX9RSSYK9zo/ENxFbeO/DcszqiYuBuY4A/dsf6VleBb+CW304q6s0k99GQDkjdIJBn8Fq58Q/Cdz4qk0eGAYiS4PnvuwUTHJ/LI/EVj3Xw+1iwtnm0K8hiuxqUlwgbgCJk2AfUKT+dOzuPW5Z0SdLbwu4bAE2jTz4/wBlZJD/AO1KtaGh03xbpcL8CfQ0Uk/3k2AD8gxpNc8C312ujQadfiG2trUWN0p/5aQnG7HucVf8a+ErnxDHYtp179iuLdypkGR+7IwQMd8E/maLMDiNXu/7L8I+HPEGG2R6nJeE/wB5XlPH4pmp/EPi+DVPDD6uZGFlLrsCWgYYzHGFLHH1DGu38SeDoNb8GL4fgcQLEiLA5GQu0Y5/DNZ9z8MtIvfDWk6LcPL5OnqxBQ43u3Vj+OT+NDQWZwfivxDDay+LtIUM819Ms8ZUZBRVO45/4APzrpW8W2tv4+0q98uVo9a0+KO3Xbzl5I8Z+gyfwrY/4Vvpsmoajczyu63VkLOMY5jXGGOfUkZq9P4JsJvEei6ruIXSrfyYocccDCn8P8KLAkzzjw1r91a+I5fBMcTlpNZW4Mo6IikSSL+afqa67w/fw6pq3h2CGRXeyS9Mig52BWEa5+oat2w8FaZYeML3xJGGN1crjaeiE/eI+uP1NTaH4P0nw9ql/f2MRWW8OXBPCjOSB6AnmhIaTOgoooqiiF5oY3w8qKf9pgKjN9ZjrdQD/toK43xH8O08R6/Lfy6pewLIqgJE2FGAB6+1ZA+DVpkiTVL1xng+aRS1Ju77Hop1XTlzm/tRj/psv+NMbW9KQZbUrMAcn98v+Nefj4MaXuBa7uz/ANt+v6VMvwb0T+N7hue8uc/pRZhdnZt4o0JQSdWs8Dr++FVZPG/hqP72tWn4NmucHwd8Oj+CQj3arMfwl8MIBm03Y9SaVmGppN8RPCaZ/wCJ1b/gG/wqrN8U/CMR/wCQoG5x8sbf4UifC/wwj7vsK5rQj8CeHY02/wBnREe60WYe8Y0nxf8ACUef9Kmb/diNQN8ZvCwXIN2fYQ//AF66f/hDtAyD/Zlvx0+WnjwnoQwf7Mtzj/Yos+4e8cc3xs8NgfJbag59oR/jVeT43aSEzFpWouew8sD+td8vhrRkOV022/74FTLoemL0sLcf9sxRyhqeYt8bWfP2fw3eN6Fv/rZrV8J/EjU/EfiO306bQJbSCUMWmYnC4UkdvbFd+mm2UYwlpCB7IKmSCKNgUjRfooFCQWZJRRRVFCd6KWigBKKWjvQAlFLRQAmKKWigBKKWigBKWiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooA//2Q==
You already have the images, the one you posted(in the imgsrc variable) is
You only need to decode it using base64 module, and save the result to a file.
To decode your image i've used this service.
To decode using Base64 you should use #strict_decode64 method:
$ cat testb64.rb
imgsrc='/9j/4AAQS... ...oooA//2Q==' #( snipped here your long variable,
# removed "data:image/jpeg;base64,"
# from the beginning )
require 'base64'
print Base64.strict_decode64(imgsrc)
$ ruby testb64.rb >img.jpg
$ xxd -p img.jpg
ffd8ffe000104a464946....
(valid JFIF header, viewable JPEG by Gwenview and Dolphin)
This should work:
require 'open-uri'
require 'base64'
require 'open-uri'
def save_image(imgsrc)
File.open("images/img1", "wb") do |fo|
fo.write(Base64.decode64(open(imgsrc).read))
end
end
It will save to the file path "images/img1" so you'll want to create separate paths for each file otherwise they'll overwrite each one.
"wb" means to open the output file using binary mode, which avoids line-end translations appropriate for your OS. Without b, Ruby will look for "\r" and "\n" and either remove or add them as necessary for a text file, which will corrupt a binary file. b avoids that step. This is documented in the IO.new description.
You can't pass
imgsrc => data:image/jpeg;base64,/9j/4AAQSkZJRgABAQEAYABgAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5PTgyPC4zNDL/2wBDAQkJCQwLDBgNDRgyIRwhMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjL/wAARCABxAHEDASIAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwD3+iiigAooooAKKKKACiiigAoNFYnibWG0a2sJF6XF/BbsfRWbB/SgDbooooAKKKKACiiigApDRmqGs6vbaJpkl/d7/KjIGEXLEk4AA7nJoA0KKgtLqG9tUuIHDxOMqw71NQAtFGaTPFIBaSjNGaYC1ynjyxOoWOkwjPOqW5JHb5s5/TH411WaY4R1G8AgEEZ7HtSEySmeYvmiPcN5G7HfFKTyB3rnDcyt8RhCpHkR6Yd4z0YyD+mPzpjOkozTd6lymRuAyRnmmieNpmhEiGRRkoGGQPcUrgS0UUUwOa8XeM7DwfFaPepJI1zJtVY+oUfeb6DI/OpfEDxz/wBh8h4ptQjPqGGx2H6gVi+KLGDVvEt1azxrIE0SV03DO1vMBBH5VVW8x4F8FXMr/vFubJTk8njY36EmpZLZ1ukmHTdAYudkVs028+gV2yf0JrK8EeNE8XxXpNq1rLbyDEbHJMbZ2t+ODVvxSRZeDdV2twY5Cc/7bc/+hGsrRbZbH4gPFGgjE2kI7ADGdjIo/maYXLvjrxPP4Y0eOeytftN5NJsji7YA3MT7AD9ap694svD4IsdT0SASX2oBRBG3Ow7S759cBWq/rca3Xi7RbZxuUwXJZcfwlQp/mPzrl/DM6pb+EopTgfbdQiAPTIMmB+WRSYX1L3iPxHro0HRf7ISNdRuYRdXJPKpGoXf/AOPOPwzTvH2ta4ltZWvh0hbtomvZ3P8ADCmMj8S36VJFa+TrtlpUzYlk0SeFQe53pn9Oa0LCAXHii9imGWg0yC3Yf7xYn/PtQGpX1bxFdXvw9h1DSUP9o6jGkNsn92V+D/3z8x/4DXHL42v18I+G9Put51G61EWtxKeyxypn8SGX9a1vCrEweHbBzkQapeEZ9kkZf0ko8SaKtpottNJAFk/4SNJo+OVUyhf1Cg0hNuxV8UarrcHxBt9Vtpyujabcw2U8QP3zIMuSPYH9Kzb6TxBefHRre1WSO0jlhebb0MI2Ek+xK1ueKLK4t/Cnim4niMYfUDLFn+JQFG7+f5Vv24A+J943/UMXP/fa0w1uc/ps2oR/EWPXHumbT9TubjT1g7IIuEYfVkb86n8MWE3/AAm8viGS6kkGqm6jWIn5USKTav6Kv61Sj1C2Tw34SAmT7Q+srhARu+aZ8/zra+Htzb6npCpuxc6XeXMTr7M7EfoR+VCBHdZoooqizE1xLDT7bUdYnKxytaeQ0jH+EbiB+bfyrz+6iefwDpflj97Y6Y1xGP8Ab8xAv6Bvzrf8bxf21r+maHISbSOCbUbhM/f8sYQH23HNUNAH2iysLLaGaXQJGVfUrKP6kVD3Ie50NtHdalr9wLq5S50W7skmitmQYUnb379CfxqrY6lbah8U51tpRJ9l06SCUjoHEkZI/DcKzvDWsND4UuLrful0zTfIyf70byqv57F/Oo/BOlppXiTTIxzLPobXE7nq8jyozEnuecfhTQHcaxf2Wj2M2q3pVI7eM5kI5AOOB9SB+leP6vJcw/D/AEnV7IlZrHVLm6T/AL/Nx+RJPsDXoXjKEalrHhvRpADb3F400ynoyxIXAPsTisXTbJNR8AyWwUNPHdXk6Rf31WV1dfxVyv8AwKh6g9WZHj3WpoJ/C/jCxB2xxrKVHeN8bx/48o/4FXa6NMs3j3X3Vso1nZsv0Ikrkxp0ep/C20gVt8tjauHH96Eu6H8vLDfVBWz4LkZvEczN/wAttFsH+pCtn/0IUkC3Mrw0SniXT4Sf9Xqd2B/4DJ/ga6b4gf8AIFsf+wpaf+jRXK6U5g8V6VMfuTa3eJ/5AKj9RXVfEA40Wx/7Cdp/6MFPoC2MD4qa8yWkmhW0XmM1ubi7ftFGOF/Etj8q2i2z4h6iw6jSc/8AjwrlfEiCe28f3zcyBre2X2VBnH65rqtu/wCIuoJ/e0nH/jy0kPqcx4K+GukXMGn+Ip5bprtLlp1TzPkVllOOP+AiovBsc+ifFnUbME/ZdQjl+X0kib+ePm+jiuw+HVysvgq3yQDHcTxn6+c+P5is5bdZvGvhrW1UK2oQu8iDoHEByfxGwf8AABTQktmd/miiiqLOH1iVIviBcO5AC+H5zknpiRCaxtCmFlf+C7hzhLm0lsyfc5YD80FT+OvD1/rfjfRktfOjt5YWiuZUyB5OcupPuMDHfNZlz4P1u1+Hdt5Ku+r2V891AinJQb+MfgAfxNTrcze5IYE0638daYM5KRzRjP8AAcr/ADGfxrc0W+gn8d6aiOv/ACABgZ9WjYD8qxZvBWtPdWt9PI0lxNZTyX+xuJJTysf0yVA/3TUngv4fX2lX3h/VLqTEkMcz3Cs3zAsoVF+gX9RSSYK9zo/ENxFbeO/DcszqiYuBuY4A/dsf6VleBb+CW304q6s0k99GQDkjdIJBn8Fq58Q/Cdz4qk0eGAYiS4PnvuwUTHJ/LI/EVj3Xw+1iwtnm0K8hiuxqUlwgbgCJk2AfUKT+dOzuPW5Z0SdLbwu4bAE2jTz4/wBlZJD/AO1KtaGh03xbpcL8CfQ0Uk/3k2AD8gxpNc8C312ujQadfiG2trUWN0p/5aQnG7HucVf8a+ErnxDHYtp179iuLdypkGR+7IwQMd8E/maLMDiNXu/7L8I+HPEGG2R6nJeE/wB5XlPH4pmp/EPi+DVPDD6uZGFlLrsCWgYYzHGFLHH1DGu38SeDoNb8GL4fgcQLEiLA5GQu0Y5/DNZ9z8MtIvfDWk6LcPL5OnqxBQ43u3Vj+OT+NDQWZwfivxDDay+LtIUM819Ms8ZUZBRVO45/4APzrpW8W2tv4+0q98uVo9a0+KO3Xbzl5I8Z+gyfwrY/4Vvpsmoajczyu63VkLOMY5jXGGOfUkZq9P4JsJvEei6ruIXSrfyYocccDCn8P8KLAkzzjw1r91a+I5fBMcTlpNZW4Mo6IikSSL+afqa67w/fw6pq3h2CGRXeyS9Mig52BWEa5+oat2w8FaZYeML3xJGGN1crjaeiE/eI+uP1NTaH4P0nw9ql/f2MRWW8OXBPCjOSB6AnmhIaTOgoooqiiF5oY3w8qKf9pgKjN9ZjrdQD/toK43xH8O08R6/Lfy6pewLIqgJE2FGAB6+1ZA+DVpkiTVL1xng+aRS1Ju77Hop1XTlzm/tRj/psv+NMbW9KQZbUrMAcn98v+Nefj4MaXuBa7uz/ANt+v6VMvwb0T+N7hue8uc/pRZhdnZt4o0JQSdWs8Dr++FVZPG/hqP72tWn4NmucHwd8Oj+CQj3arMfwl8MIBm03Y9SaVmGppN8RPCaZ/wCJ1b/gG/wqrN8U/CMR/wCQoG5x8sbf4UifC/wwj7vsK5rQj8CeHY02/wBnREe60WYe8Y0nxf8ACUef9Kmb/diNQN8ZvCwXIN2fYQ//AF66f/hDtAyD/Zlvx0+WnjwnoQwf7Mtzj/Yos+4e8cc3xs8NgfJbag59oR/jVeT43aSEzFpWouew8sD+td8vhrRkOV022/74FTLoemL0sLcf9sxRyhqeYt8bWfP2fw3eN6Fv/rZrV8J/EjU/EfiO306bQJbSCUMWmYnC4UkdvbFd+mm2UYwlpCB7IKmSCKNgUjRfooFCQWZJRRRVFCd6KWigBKKWjvQAlFLRQAmKKWigBKKWigBKWiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooA//2Q==
as the URL for an image, as that isn't a URL. Both OpenURI and Net::HTTP expect a URL to the image, which they will then request and read the resulting response, returning the data back to your code. You'd need to do a Base64 decode against that data, which will result in a binary string in memory, which you can then write to a file opened in binary mode.
Related
I want to read data from csv files with two possible encodings (UTF-8 and ISO-8859-15). I mean different files with different encodings. Not the same file with two encodings.
Now I can only read data correctly from a utf-8 encoding file. Can I just implement this by adding an extra option? For example . encoding: 'ISO-8859-15'
What i have:
def csv
file = File.open(file.tempfile)
CSV.open(file, csv_options)
end
private
def csv_options
{
col_sep: ";",
headers: true,
return_headers: false,
skip_blanks: true
}
end
Once you know what encoding your file has, you can pass inside the CSV options i.e.
external_encoding: Encoding::ISO_8859_15,
internal_encoding: Encoding::UTF_8
(This would establish, that the file is ISO-8859-15, but you want the strings internally as UTF-8).
So the strategy is that you decided first (before opening the file), what encoding you want, and then use the appropriate option Hash.
Answer (updated)
New update guys, I chose to remove this because I read that this code, and basically this whole thing I had created, was not good enough and if the bot would get larger, it would cause problems because it would overload the .JSON file. So I did this:
Main File:
def get_prefix(client, message):
with open('prefixes.json', 'r') as file:
prefixes = json.load(file)
if str(message.guild.id) in prefixes:
return prefixes[str(message.guild.id)]
else:
return "/"
client = commands.Bot(command_prefix = get_prefix)
Prefixes File: (INSIDE A COG)
#commands.Cog.listener()
async def on_guild_remove(self, guild):
with open('prefixes.json', 'r') as f:
prefixes = json.load(f)
prefixes.pop(str(guild.id))
with open('prefixes.json', 'w') as f:
json.dump(prefixes, f, indent=4)
So before this "Update" the bot would just load the Default prefix to the .JSON file. Now what I have done is that the Get_Prefix will check if the guild's ID is on the .JSON File, and if in any case is not in there it will use the / as a Default prefix. This will help because the code will not have to store every single server to the .JSON File which would cause problems in the future if the bot had the chance to get bigger and more well known.
So then in the Prefix File (can be also used in your Main File but you will have to remove the Self that I have used.) I removed the code which the bot would have to add the prefix to the .JSON file and I only let the part that will remove the Guild's Custom Prefix if it is on the .JSON file. That's pretty much it, if you have any more questions I can help you with this just comment in this question!
Question
I'm struggling to find a solution in a large problem I have with my bot. Basically I'm using a per-server-prefix code, which I found as a tutorial in the Internet. It's a pretty basic one, I believe that there are more advanced codes than this one but that's not the point at the moment. Before I begin with my problem I want to explain to you how my code works, Basically whenever the Client joins a new server it will save it's ID in a JSON File and it will give it a Default Prefix.
import discord
import json
import asyncio
import os
from discord.ext import commands
class Prefixes(commands.Cog):
def __init__(self, client):
self.client = client
#commands.Cog.listener()
async def on_guild_join(self, guild):
with open('prefixes.json', 'r') as f:
prefixes = json.load(f)
prefixes[str(guild.id)] = '/'
with open('prefixes.json', 'w') as f:
json.dump(prefixes, f, indent=4)
#commands.Cog.listener()
async def on_guild_remove(self, guild):
with open('prefixes.json', 'r') as f:
prefixes = json.load(f)
prefixes.pop(str(guild.id))
with open('prefixes.json', 'w') as f:
json.dump(prefixes, f, indent=4)
then if you wish to change the prefix to something else you can use the command /prefix and it will change to the new prefix while the client will save the new prefix in the JSON File.
#commands.command()
#commands.has_permissions(manage_roles=True, ban_members=True)
async def prefix(self, ctx, prefix=None):
if prefix == None:
try:
x = ""
pfp = self.client.user.avatar_url
prefix = discord.Embed(title=x, description=f"My prefix for **{ctx.guild.name}** is `{ctx.prefix}`. If you want to find out more information about me type `{ctx.prefix}help`.", color = 0x456383)
prefix.set_footer(text=f"ChizLanks", icon_url=pfp)
await ctx.channel.send(embed = prefix)
return
except discord.Forbidden:
return await ctx.channel.send(f"My prefix for **{ctx.guild.name}** is `{ctx.prefix}`. If you want to find out more information about me type `{ctx.prefix}help`")
else:
with open('prefixes.json', 'r') as f:
prefixes = json.load(f)
prefixes[str(ctx.guild.id)] = prefix
with open('prefixes.json', 'w') as f:
json.dump(prefixes, f, indent=4)
await ctx.channel.send(f'Server Prefix has changed to `{prefix}`')
That's pretty much the code, I also use some errors for the command but that is not needed at the moment. My problem is that if the client is offline the code is basically not running, that means if the bot gets invited to a new server, the client will not be able to save the prefix to the JSON File and this will cause problems to the server, because when the client will get back online, it will not have any prefix and that means that they will not be able to use any commands.
Now how can that be fixed? I already have an idea of how it will work. I will probably need to use on_ready event and that event will search if the servers that the client has joined are in the JSON File it will use them, otherwise it will create a a new prefix (The Default One) and it will save it to the JSON File. That's my idea but I need some help because I do not have any idea if this is even possible. How might this work?
You could add a default prefix that is recognized by the bot if there is no prefix found for the server. That way, when the very first command is sent in a server, the bot can recognize it even though it wasn't added to the prefix file. Then, when it runs that default prefix, the server could then be registered in the JSON file and everything will be normal from there.
Here is a Ruby question guys. So need to parse through the html file and catch urls and emails can't come up with proper regex expression. Tried 100+ regexes and all the times I cash something else with the url.
File.open("/Desktop/file.html").each_line do |line|
if line.split("href=\"") =~ /???/
puts line
end
end
# I can use line.split("href=\"") so each new line will start with url =>
(https://www.facebook.com/students">
The question is what regex can I use to catch everything from https to the end of the url which ends with (") (there could be one or more samples of same url so {1,2} is needed
Try this
file = File.open('filename_path')
links = file.read().scan(/href=\"(?<url>.*?)\"/)
you get links in array
it also works if you remove ?<url> from above(it's just named capture group)
EDIT: Now that the problem is solved, I realize that it had more to do with properly reading/writing byte-strings, rather than HTML. Hopefully, that will make it easier for someone else to find this answer.
I have an HTML file that's poorly formatted. I want to use a Python lib to just make it tidy.
It seems like it should be as simple as the following:
import sys
from lxml import etree, html
#read the unformatted HTML
with open('C:/Users/mhurley/Portable_Python/notebooks/View_Custom_Report.html', 'r', encoding='utf-8') as file:
#write the pretty XML to a file
file_text = ''.join(file.readlines())
#format the HTML
document_root = html.fromstring(file_text)
document = etree.tostring(document_root, pretty_print=True)
#write the nice, pretty, formatted HTML
with open('C:/Users/mhurley/Portable_Python/notebooks/Pretty.html', 'w') as file:
#write the pretty XML to a file
file.write(document)
But this chunk of code complains that file_lines is not a string or bytes-like object. Okay, it makes sense that the function can't take a list, I suppose.
But then, it's 'bytes' not a string. No problem, str(document)
But then I get HTML that's full of '\n' that are not newlines... they're a slash followed by an en. And there are no actual carriage returns in the result, it's just one long line.
I've tried a number of other weird things like specifying the encoding, trying to decode, etc. None of which produce the desired result.
What's the right way to read and write this kind of (is non-ASCII the right term?) text?
You are missing that you get bytes from tostring method from etree and need to take that into account when writing (a bytestring) to a file. Use the b switch in the open function like this and forget about the str() conversion:
with open('Pretty.html', 'wb') as file:
#write the pretty XML to a file
file.write(document)
Addendum
Even though this answer solves the immediate problem at hand and teaches about bytestrings, the solution by Padraic Cunningham is the cleaner and faster way to write lxml etrees to a file.
This can be done all using lxml in a couple of lines of code without ever needing to use open, the .write method is exactly for what you are trying to do:
# parse using file name which is the also the recommended way.
tree = html.parse("C:/Users/mhurley/Portable_Python/notebooks/View_Custom_Report.html")
# call write on the tree
tree.write("C:/Users/mhurley/Portable_Python/notebooks/Pretty.html", pretty_print=True, encoding="utf=8")
Also file_text = ''.join(file.readlines()) is exactly the same as file_text = file.read()
I have a rake task that populates my database from a CSV file:
require 'csv'
namespace :import_data_csv do
desc "Import teams from csv file"
task import_data: :environment do
CSV.foreach(file, :headers => true) do |row|
#various import tasks
This had been working properly, but with a new CSV file, I'm getting the following error on the 6th row of the CSV file:
Invalid byte sequence in UTF-8
I have looked through the row and can't seem to find any irregular characters.
I've also attempted a couple other fixes recommended on stackoverflow:
- Changing the CSV.foreach to:
reader = CSV.open(file, "r")
reader.each do |row|
And changing:
CSV.foreach(file, headers => true) do |row|
to:
CSV.foreach(file, encoding: "r:ISO-8859-1", :headers => true) do |row|
None of these seem to correct the issue.
Suggestions?
I solved this by saving the file as a MDOS CSV, instead of the standard CSV file or the Windows CSV format.
The answer for me was to take the CSV file and save it to a text file. Then replace the tabs with commas. Then save the file as UTF-8 encoded. Finally, change the .txt to .csv and make sure it works in Excel BUT DON'T save it in Excel. Just close it when you see it looks correct. Then upload it.
A long non-programatic solution, but for my purposes it's sufficient.
Source is here: https://help.salesforce.com/apex/HTViewSolution?id=000003837&language=en_US