In a project, we use 2 IDEs. The project contains hundreds files of code, and hundreds special files of JSON format which constantly get reread and rewritten by these IDEs. While we used single IDE, it's not a problem, files always get written the same way. Unfortunately, different IDEs save JSON with different ordering which leads to dozens of changes for GIT and uselessly overwhelmed diff. These files are important and must not be excluded by GitIgnore, but they rarely get changed, and this probably can be handled manually.
So, is there a terminal command to quickly undo/unselect changes for specific file extension? Or, maybe it is possible for GIT to track changes of JSONs without considering the order?
I also had an idea to use custom script for reordering the JSONs, but it would consume too much CPU, and also lead to rereading by an IDE which is also bad.
Update
I found the following command from another SO question:
git checkout main -- $(git ls-files -- "*.yy")
This workaround isn't handy but basically solves the problem. If anybody knows how to make GIT ignore JSON ordering, it would be great!
One way to temporarily ignore changes to the json files is to tell git to assume they haven't changed:
git update-index --assume-unchanged file-to-ignore.json
And only when you want to commit, tell git to really look at the file again:
git update-index --no-assume-unchanged file-to-ignore.json
Another option would be to use a pre-commit-hook to sort the json only when committing.
i'd make a git pre-commit hook to make sure all JSONs are always formatted the same way, for example in .git/hooks/pre-commit put
#!/bin/sh
php git/precommit_hook.php
exit $?
and if you're on a unix-system, make sure pre-commit is chmod +x .git/hooks/pre-commit
and in git/precommit_hook.php put
<?php
declare (strict_types = 1);
if(PHP_VERSION_ID < 70300) {
fwrite(STDERR, "PHP 7.3 or higher is required to run this script");
exit(1);
}
$changed_files = explode("\x00", rtrim(shell_exec("git diff --name-only --cached -z"), "\x00"));
foreach ($changed_files as $file) {
if(!file_exists($file)) {
// File was deleted, skip it
continue;
}
$ext = pathinfo($file, PATHINFO_EXTENSION);
if ($ext === "json") {
$json = json_decode(file_get_contents($file), true);
if (json_last_error() !== JSON_ERROR_NONE) {
fwrite(STDERR, "JSON Error: " . json_last_error_msg() . " in $file, will not format it\n");
continue;
}
$json = json_encode($json, JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE | JSON_THROW_ON_ERROR);
file_put_contents($file, $json, LOCK_EX);
}
}
now all *.json files will be committed with the PHP json formatters JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE | JSON_THROW_ON_ERROR
no matter what IDE you use :)
Related
I am extracting prosody features from an audio file while using Opensmile using Windows version of Opensmile. It runs successful and an output csv is generated. But when I open csv, it shows some rows that are not readable. I used this command to extract prosody feature:
SMILEXtract -C \opensmile-3.0-win-x64\config\prosody\prosodyShs.conf -I audio_sample_01.wav -O prosody_sample1.csv
And the output of csv looks like this:
[
Even I tried to use the sample wave file given in Example audio folder given in opensmile directory and the output is same (not readable). Can someone help me in identifying where the problem is actually? and how can I fix it?
You need to enable the csvSink component in the configuration file to make it work. The file config\prosody\prosodyShs.conf that you are using does not have this component defined and always writes binary output.
You can verify that it is the standart binary output in this way: omit the -O parameter from your command so it becomesSMILEXtract -C \opensmile-3.0-win-x64\config\prosody\prosodyShs.conf -I audio_sample_01.wav and execute it. You will get a output.htk file which is exactly the same as the prosody_sample1.csv.
How output csv? You can take a look at the example configuration in opensmile-3.0-win-x64\config\demo\demo1_energy.conf where a csvSink component is defined.
You can find more information in the official documentation:
Get started page of the openSMILE documentation
The section on configuration files
Documentation for cCsvSink
This is how I solved the issue. First I added the csvSink component to the list of the component instances. instance[csvSink].type = cCsvSink
Next I added the configuration parameters for this instance.
[csvSink:cCsvSink]
reader.dmLevel = energy
filename = \cm[outputfile(O){output.csv}:file name of the output CSV
file]
delimChar = ;
append = 0
timestamp = 1
number = 1
printHeader = 1
\{../shared/standard_data_output_lldonly.conf.inc}`
Now if you run this file it will throw you errors because reader.dmLevel = energy is dependent on waveframes. So the final changes would be:
[energy:cEnergy]
reader.dmLevel = waveframes
writer.dmLevel = energy
[int:cIntensity]
reader.dmLevel = waveframes
[framer:cFramer]
reader.dmLevel=wave
writer.dmLevel=waveframes
Further reference on how to configure opensmile configuration files can be found here
My purpose is to analyse web-application logs, use mysql as database. First, i filtered some useless information use awk to generate a filted-log, then i apply LOAD DATA import this log to mysql.
My problem is : those original logs generate every 10mins, every day. How can i generate filted-logs once new web-application logs was generated? After new filted-logs generated, how can i import those files to mysql automatically?
the original logs:
20150414/0900.log
20150414/0910.log
I´ve tried to create a little script that will easy explain the way to do it. There you have an awk that controls all the readFiles. If the number of read files is bigger when a new read is done, the system will parse de name and save it in "readFiles" file which will be checked in the awk to assure the file was not read before.
Please, check your system will not erase old logs, and be careful of splitting readed control files, or creating new ones each day to avoid very big files.
//this will give you the today datae
date +%Y%m%d
This is the code:
echo "x" > readFiles
lastnum=0
num=0
count=0
while true
do
echo "LOOKING FOR NEW FILES. LASTCOUNT="$lastcount
count=`ls ./2015*/*.log | wc -l`
echo $count
if [ $count -gt $lastnum ]
then
lastnum=$count
`ls ./2015*/*.log | awk -F"/" 'BEGIN {
while(( getline < "readFiles") > 0 ) {
readedFiles[$0]
}}
{if(!($0 in readedFiles)){print $0}}
'`>> readFiles
echo "WAITING RESTART"
sleep 10
else
echo "NO NEW FILES FOUND"
sleep 10
fi
done
Instead of writing script to monitor logs. I use inotify-tools to trigger scripts on filesystem events, just few lines get things done.
NOW=$(date +"%Y%m%d")
while true ;
do
inotifywait -r -e create,move /rsynclog/logs/$NOW && \
/rsynclog/logs/generate.sh
done
#!/bin/bash
read()
{
count=0
cat localfile |
while read line
do
FILE[$((count += 1))]="$line"
done
}
read
for((i=0;i<${#FILE[#]});i++)
do
echo ${FILE[i]}
done
The result of echo is whole blank. Is there any way to get the FILE array?
You posted this under ash, the Almquist shell, but you are using bash, the Bourne-Again SHell.
One problem is the pipe. When you run a pipe in bash, each side runs in its own sub-shell, and any variables are local to it. The correct mechanism is to redirect into the loop.
Another problem is that your function is called read, which is a shell-builtin (you use it!). A good idea is to use a naming convention, like an f_ prefix, so you don't get these name collisions.
Another issue you have is that the syntax of your second for loop is wrong in several ways. Here is a corrected version:
#!/bin/bash
f_read()
{
count=0
while read line
do
FILE[$((count += 1))]="$line"
done < localfile
}
f_read
for ((i=0;i<${#FILE[#]};i++))
do
echo ${FILE[i]}
done
I'm using Mercurial (specifically TortoiseHg on Windows) to do version control of VBA code. Anybody who's tried this knows that VBA changes the case of every variable throughout a project whenever any declaration of that variable is changed anywhere in the project (regardless of scope). It makes version control a nightmare.
I would like to ignore case changes in my source code when performing diffs. What is the easiest way to do this? (some option for diff that I'm missing, an external diff utility, something else?)
NOTE: I am not talking about dealing with 'case-insensitive filenames' (yes, I'm talking to you Google...)
You can do that when diffing for your on-screen consumption using the ExtDiff Extension.
[extensions]
hgext.extdiff =
[extdiff]
# add new command that runs GNU diff(1) in case-insensitive mode
cmd.mydiff = diff
opts.mydiff = -i
Then you'd run hg mydiff from the command line. That, of course, requires you have a diff binary installed be it gnu's or other.
However, that's not going to be as helpful as you might like because internally, of course, Mercurial can't ignore case -- it's taking the cryptographic hash of the file contents, and those don't allow for wiggle room. So if you get this set up you'll do hg mydiff, and see no changes, and then do hg commit and see changes all over the place.
So you can make this work on-screen, but not fundamentally.
One option would be to find a visual basic code-cleaner, similar to indent for C-like languages, that normalizes variable case and run that in a mercurial commit hook. Then at least all the code going into source control will be consistent and you can diff across revisions accurately.
If you are okay with having your code in all lower-case, say, then you could employ the encode/decode hooks for this. It would work like this:
[encode]
*.vba = tr A-Z a-z
This will encode the file content in lower-case whenever you do a commit. The diffs are also computed based on the encoded (repository) version of the files.
Consider a file that contains
hello
Changing it in your working copy to
Hello World
will give a diff of
% hg diff
diff --git a/a.txt b/a.txt
--- a/a.txt
+++ b/a.txt
## -1,1 +1,1 ##
-hello
+hello world
Notice how the capital "H" and "W" has been ignored.
I don't really know anything about VBA code, so I'm not 100% sure this solution works for you. But I hope it can be a starting point.
One drawback is that you'll need to set this encode rule for all your repositories. The reposettings extension can help you here.
Here's the solution I have settled on. It is far from ideal, but better than the other alternatives I've considered.
I created an Autohotkey script that does the following:
reverts MS Access files in a repository with detected changes (to .orig files)
reads in the .orig file (the one with the changes)
reads in the existing file (the one already in the repository)
converts the text of both files to lower case
compares the lower case contents of the files
if the files still differ, the .orig file is restored so it may be committed to the repository
if the files are the same (i.e., they differ only in case, the .orig file is deleted because we don't care about those changes)
For files that have actual changes that we care about, I still see the case changes that were made as well. If that results in a lot of noise, I open the file in a comparison tool that allows case-insensitive compares (e.g., kdiff).
It's not a perfect solution, but it removes about 90% of the frustration for me.
Here's my script. Note that the script includes another Autohotkey script, ConsoleApp.ahk, which provides a function named, ConsoleApp_RunWait(). This is a 3rd party script that no longer works very well with 64-bit AHK, so I'm not including it as part of my answer. Any AHK function that executes a command line and returns the output as a string will suffice.
; This script checks an MS Access source directory and reverts all files whose only modifications are to the
; case of the characters within the file.
#Include %A_ScriptDir%\ConsoleApp.ahk
#NoEnv ; Recommended for performance and compatibility with future AutoHotkey releases.
SendMode Input ; Recommended for new scripts due to its superior speed and reliability.
SetWorkingDir %A_ScriptDir% ; Ensures a consistent starting directory.
; Allow for custom path to hg (support for moving to TortoiseHg 2.0)
IniRead hg, %A_ScriptDir%\LocalSettings\Settings.cfg, TortoiseHg, hg_path, hg
if 0 < 1 ; The left side of a non-expression if-statement is always the name of a variable.
{
MsgBox Usage:`n`HgIgnoreCase DirectoryWithFilesToScrub
ExitApp
}
SrcDir = %1%
StringReplace SrcDir, SrcDir, ", , All
StringRight test, SrcDir, 1 ; add trailing slash if necessary
ifnotequal test, \
SrcDir = %SrcDir%\
RestoreOriginals(SrcDir)
RevertCaseChangeModifiedFiles(SrcDir)
RevertCaseChangeModifiedFiles(SrcDir) {
global hg
includes = -I "*.form" -I "*.bas" -I "*.report" -I "*.table"
cmdline = %hg% revert --all %includes%
;Don't revert items that have been removed completely
Loop 3
{
Result := ConsoleApp_RunWait(hg . " status -nrd " . includes, SrcDir)
If (Result)
Break
}
Loop parse, Result, `n, `r
{
if (A_LoopField)
cmdline = %cmdline% -X "%A_LoopField%"
}
Result =
;msgbox %cmdline%
;revert all modified forms, reports, and code modules
Loop 3
{
Result := ConsoleApp_RunWait(cmdline, SrcDir)
If (Result)
Break
}
;MsgBox %Result%
Loop parse, Result, `n, `r
{
StringLeft FileStatus, A_LoopField, 9
If (FileStatus = "reverting")
{
StringMid FName, A_LoopField, 11
FullPath = %SrcDir%%FName%
ToolTip Checking %FullPath%
RestoreIfNotEqual(FullPath, FullPath . ".orig")
}
}
ToolTip
}
RestoreIfNotEqual(FName, FNameOrig) {
FileRead File1, %FName%
FileRead File2, %FNameOrig%
StringLower File1, File1
StringLower File2, File2
;MsgBox %FName%`n%FNameOrig%
If (File1 = File2)
FileDelete %FNameOrig%
Else
FileMove %FNameOrig%, %FName%, 1
}
RestoreOriginals(SrcDir) {
Loop %SrcDir%*.orig
{
;MsgBox %A_LoopFileLongPath%`n%NewName%
NewName := SubStr(A_LoopFileLongPath, 1, -5)
FileMove %A_LoopFileLongPath%, %NewName%, 1
}
while FileExist(SrcDir . "*.orig")
Sleep 10
}
So the utility Diff works just like I want for 2 files, but I have a project that requires comparisons with more than 2 files at a time, maybe up to 10 at a time. This requires having all those files side by side to each other as well. My research has not really turned up anything, vimdiff seems to be the best so far with the ability to compare 4 at a time.
My question: Is there any utility to compare more than 2 files at a time, or a way to hack diff/vimdiff so it can do multiple comparisons? The files I will be comparing are relatively short so it should not be too slow.
Displaying 10 files side-by-side and highlighting differences can be easily done with Diffuse. Simply specify all files on the command line like this:
diffuse 1.txt 2.txt 3.txt 4.txt 5.txt 6.txt 7.txt 8.txt 9.txt 10.txt
Vim can already do this:
vim -d file1 file2 file3
But you're normally limited to 4 files. You can change that by modifying a single line in Vim's source, however. The constant DB_COUNT defines the maximum number of diffed files, and it's defined towards the top of diff.c in versions 6.x and earlier, or about two thirds of the way down structs.h in versions 7.0 and up.
diff has built-in option --from-file and --to-file, which compares one operand to all others.
--from-file=FILE1
Compare FILE1 to all operands. FILE1 can be a directory.
--to-file=FILE2
Compare all operands to FILE2. FILE2 can be a directory.
Note: argument name --to-file is optional.
e.g.
# this will compare foo with bar, then foo with baz .html files
$ diff --from-file foo.html bar.html baz.html
# this will compare src/base-main.js with all .js files in git repo,
# that has 'main' in their filename or path
$ git ls-files :/*main*.js | xargs diff -u --from-file src/base-main.js
Checkout "Beyond Compare": http://www.scootersoftware.com/
It lets you compare entire directories of files, and it looks like it runs on Linux too.
if your running multiple diff's based off one file you could probably try writing a script that has a for loop to run through each directory and run the diff. Although it wouldn't be side by side you could at least compare them quickly. hope that helped.
Not answering the main question, but here's something similar to what Benjamin Neil has suggested but diffing all files:
Store the filenames in an array, then loop over the combinations of size two and diff (or do whatever you want).
files=($(ls -d /path/of/files/some-prefix.*)) # Array of files to compare
max=${#files[#]} # Take the length of that array
for ((idxA=0; idxA<max; idxA++)); do # iterate idxA from 0 to length
for ((idxB=idxA + 1; idxB<max; idxB++)); do # iterate idxB + 1 from idxA to length
echo "A: ${files[$idxA]}; B: ${files[$idxB]}" # Do whatever you're here for.
done
done
Derived from #charles-duffy's answer: https://stackoverflow.com/a/46719215/1160428
There is a simple an good way to do this = GREP.
Depending on the size of the text you can copy and paste it, or you can redirect the input of the file to the grep command. If you make a grep -vir /path to make a reverse search or a grep -ir /path. This is my way for certification exams.