Merge CSV files keeping one header

Merge CSV files keeping one header

 

1 Kudos

There are typical situations in life that require lots of engineering to be addressed. Other situations don’t. This is exactly one of those commands you may want to learn by heart but, for those who can’t, that’s why blogs exist.

Suppose you have collected statistics for several days and you have lots of files having the same structure, e.g.:

cola,colb,colc
1,2,3
1,3,5
4,3,1
...

and you want to create one big file that keeps the header from just the first one and appends all the files so that you can quickly load it into your favourite tool for further analysis.
Here is the trick:

awk 'FNR==1 && NR>1{next;}{print}' *.csv > bigfile

Explanation:
FNR is the number of records read by awk (per file), NR is the number of lines read overall. Therefore, the condition FNR==1 && NR>1 evaluates to true only if the line being evaluated is the first line of the file and we already read at least one line overall (so it is not the first line we read). What happens when it is true? {next;}, hence the line gets ignored.

Warning
I am pretty sure that some of you just came here, copied&paste the line above, tested and found out that the command never ends. Well, if this is the case the file you generated is probably taking all of your free disk space. Yummie. Why? Probably because the output file already existed before and now it matches the wildcard condition of awk. So, make sure that the output file doesn’t exist when executing the command or, at least, doesn’t end in your shell expansion.

May 19, 2015 0 comments Read More
OpenAFS and Mac OS X Yosemite

OpenAFS and Mac OS X Yosemite

 

60 Kudos

Long story short: Kerberos on the Mac, starting with Yosemite, does not support anymore weak ciphers (such as DES).

AFS, on the other hand, works with DES.. therefore.. no go. The best thing to do would be to try to migrate OpenAFS to support different ciphers but this requires maintenance on the AFS server, risking damage and data loss.

Therefore this (quite ugly, but still..) solution seems to overcome the limitation of the Kerberos installation provided by default with Mac OS X.

Step 1: System cleanup

If you have installed OpenAFS or already configured kerberos on your machine, uninstall everything and delete /Library/Preferences/edu.mit.Kerberos.

Please reboot.

Step 2: Install heimdal kerberos

Download and install this: http://www.h5l.org/dist/src/heimdal-1.5.3.dmg

This is the heimdal kerberos, the vanilla version. Therefore, this supports the aforementioned weak DES cipher.

Step 3: Install OpenAFS for Yosemite

As may already know, there is no official OpenAFS version for Yosemite. I compiled and uploaded one for you. You can download it here: https://dl.dropboxusercontent.com/u/355313/openafs/OpenAFS-1.6.10-2-gb9a15b-dirty-Yosemite.dmg

Download, open and install it.

Step 4: Configure ’em all

First of all, let’s configure Kerberos. You should already have the configuration for Kerberos. Make sure you have it in the correct path that, on Yosemite, is /etc/krb5.conf. Then make sure you add the allow_weak_crypto = true line to the libdefaults section.

[libdefaults]
...
allow_weak_crypto = true
...

Then configure OpenAFS.

  • Configure AFS by editing the ThisCell and CellServDB files accordingly. Don’t reboot now.
  • OpenAFS requires a kernel extensions to work properly. Unfortunately (yes, again), unsigned kernel extensions cannot be loaded on boot in Yosemite. However, this problem can be solved by using modifying the boot parameter of the kernel:
    $> sudo cp -r /private/var/db/openafs/etc/afs.kext /Library/Extensions
    $> sudo nvram boot-args="kext-dev-mode=1"
    
  • Now, reboot the mac
  • When everything is restored, make sure you apply the necessary settings and add AFS icon to the menu bar for quicker access:
    • Go to System Preferences > OpenAFS.
    • AFS Menu: checked
    • Backgrounder: checked
    • Use aklog: checked

Step 5: How to connect

Each time you want to use AFS, you must do the following:

  • open Terminal.app
  • issue
    $> /usr/heimdal/bin/kinit <kerberos username here>
    $> aklog
    

Everything should be working.

If you read this guide and something didn’t work, make sure you followed each step in the precise order they are written. If something is still not working properly, just drop a line in the comments and we will try to sort it out.

Update

After upgrading Yosemite to 10.10.3 or, in general, after every system update I noticed that I have to reinstall OpenAFS or, at least, re-issue the nvram command to let unsigned kernel extensions to be loaded again.

December 26, 2014 11 comments
Convert swiss national grid coordinates (CH1903) to WGS1984

Convert swiss national grid coordinates (CH1903) to WGS1984

 

5 Kudos

The World Geodetic System (or WGS) is a worldwide standard for the cartography. The well-known concepts (such as latitude and longitude, for example) are part of this standard.

Yet, in Switzerland, there is a different national standard that is called CH1903 (or Landesvermessung 1903, LV03).

Implementations, in the most widespread programming languages, to convert coordinates from the Swiss standard to WGS1984 are provided by swisstopo. However, the Python one is missing. You can grab it here.

Click here to get the code!

October 13, 2014 0 comments Read More
[HOW-TO] Homebrew: remove a formula and all its dependencies

[HOW-TO] Homebrew: remove a formula and all its dependencies

 

22 Kudos

Since homebrew does not officially support an automated way to do that, I created a small zsh function to remove a formula and all its dependencies.

I use zsh on the Mac so the function works with it. Feel free to adjust it if you are using bash or something else.

brew-remove-with-deps() {
  formula="$*"

  if [ "x$formula" = "x" ]
  then
     echo "Invalid empty parameter"
  else
     echo "Removing" "$formula" "and all its deps.."
     brew rm $formula
     brew rm $(join <(brew leaves) <(brew deps "$formula"))
  fi
}

And, just for testing purposes:

elbryan@snowbox ~ % brew-remove-with-deps sloccount
Removing sloccount and all its deps..
Uninstalling /usr/local/Cellar/sloccount/2.26...
Uninstalling /usr/local/Cellar/md5sha1sum/0.9.5...

Et voilà.

References:
[1] – Stackoverflow – Uninstall / remove a Homebrew package including all its dependencies

April 16, 2013 0 comments Read More
[HOW-TO] Activate next UITextField in UITableView (iOS)

[HOW-TO] Activate next UITextField in UITableView (iOS)

 

35 Kudos

Update: After that Matteo pointed out the fact that I had everything to be able to access to each UITableViewCell’s instances, I decided to update this blogpost accordingly.

One of the most convenient automations that a developer should build into his applications is the capability of moving to the next (text)fields in a form.

Unfortunately, this automation is not eased by Apple in its frameworks and, thus, require the developer to provide an extra-effort on his side to make this to happen.

Read more…

February 8, 2013 4 comments Read More