Convert log files to base64 and upload it using Curl to Github - json

Question

I am trying to upload a file to Github using its API.
Following code works, but only with smaller size content which is approx less than 1MB.
tar -czvf logs.tar.gz a.log b.log
base64_logs=$(base64 logs.tar.gz | tr -d \\n)
content_response=$(curl \
-X PUT \
-u :"$GIT_TOKEN" \
-H "Accept: application/vnd.github.v3+json" \
"$content_url" \
-d '{"message": "Log files", "content": "'"$base64_logs"'"}')
For content that is a bit large, I get the following error:
/usr/bin/curl: Argument list too long
Now, there is already a question on SO about this error message, and it says that to upload a file directly. See here: curl: argument list too long
When I try this, I get a problem parsing JSON error message.
tar -czvf logs.tar.gz a.log b.log
base64_logs=$( base64 logs.tar.gz | tr -d \\ ) > base64_logs.txt
content_response=$(curl \
-X PUT \
-u :"$GIT_TOKEN" \
-H "Accept: application/vnd.github.v3+json" \
"$content_url" \
-d '{"message": "Log files", "content": #base64_logs.txt}')
Can anyone point me out where I am making mistake here? Thanks!

Answer 1

Use the base64 command rather than the #base64 filter from jq, because the later can only encode textual data and not binary data as from a .gz archive.
Pipe the base64 stream to jq to format it into a JSON data stream.
Curl will read the JSON data stream and send it.
# use base64 to encode binary data and -w0 all in one-line stream
base64 --wrap=0 logs.tar.gz |
# JSON format from raw input
jq \
--raw-input \
--compact-output \
'{"message": "Log files", "content": . }' |
# Pipe JSON to curl
curl \
--request PUT \
--user ":$GIT_TOKEN" \
--header "Accept: application/vnd.github.v3+json" \
--header 'Content-Type: application/json' \
--data-binary #- \
--url "$content_url"