The Accidental Rubyist

invalid byte sequence in UTF-8

Automating export of Firefox 3 bookmarks and history from command-line/cron

with 7 comments

Firefox 2 used a bookmarks.html file for bookmarks. History was in a mork file that was incomprehensible. Firefox 3 uses SQLite for both these. Export manually is simple from the Bookmarks-Organize Bookmarks menu option. This page gives us info on how to automate bookmarks export using about:config, however this export happens:


  1. only for bookmarks, not history

  2. only when Firefox is shutdown


So the only option I had was to access the database via python or ruby. Both have packages for that, the python one comes with python (no install) so i tried that and got “database locked”. Googled but no one had an answer for that.

Finally, I just copied the file (hint courtesy), places.sqlite to places1.sqlite and was able to open (pls note that FF3 is still running).

It seems I already have a program called “sqlite3” on my os x. Loading it on the backup file showed me that I had various output formats such as html, csv, list etc. I could output to a file … same stuff as any other sql client – if you’ve used mysql, you’d be very familiar.

Now automating the output of the history and bookmarks without shutting firefox, and without user intervention in any format is as simple as:


cd /Users/john/Library/Application\ Support/Firefox/Profiles/8j7l8qg8.default
cp places.sqlite places1.sqlite
sqlite3 'places1.sqlite' <<!
.dump html
.output moz_places.html
select url,title from moz_places;
!

This may be run as a cronjob. Be sure to change the path in the cd command to your user, and edit the dump to list or csv as required. The output would go into some work folder for other applications to access easily. The existing location has a space which can cause bash programs to gripe (wish they would make life easier for us command-line junkies).

Bookmarks are stored in the moz_bookmarks file with title but not url. To get the url you would link to moz_places with the fk field as:


select a.url, b.title from moz_places a, moz_bookmarks b where a.id=b.fk and b.title like '%zsh%';

in order to dump all rows, just give:


select a.url, b.title from moz_places a, moz_bookmarks b where a.id=b.fk;

Before processing this text you may wish to filter out pages your other apps do not need such as google searches, your own websites/blogs, pagead urls etc. Also you may want to remove duplicates with the “sort -u” command.

Hope this helps.


# join pairs of lines side-by-side (like ‘paste’)
$ cat | ruby -pe ‘$_ = $_.chomp + ” ” + gets if $. % 2’

Advertisements

Written by totalrecall

September 3, 2008 at 11:33 am

Posted in being_geeky, unix

7 Responses

Subscribe to comments with RSS.

  1. Thank you for the information. However I couldn’t get this solution to import back into Firefox (the entire point) and wound up going with the about:config trick after spending quite a bit of time trying to get this to work well.

    designatevoid

    December 1, 2008 at 8:58 pm

  2. um….am I the only one that has a daily backup of my bookmarks in .json format located in my profile under /bookmarkbackups? This file will import into firefox. It’s a good way to have the same bookmarks available on other computers. The only drawback is that the bookmarks are not added….they replace all your existing bookmarks on that copy of your profile…no biggie if you have one main computer that you try to replicate out to others.

    #!/bin/bash
    for file in *.json
    do
    cp -u $file /place/to/copy/bookmarks/
    done

    mrglasshalfempty

    August 27, 2009 at 6:59 pm

  3. […] with converting the SQlite and .plist files came from here and here. Help with getting the .plist files readable came from […]

  4. Ok, a little late for this thread; however, just do a database dump of places.

    echo ‘.dump’ | sqlite3 places.sqlite > places.sqlite.sql

    Use sqlite to create a new db and read the file. The output of the above command is pure sql. Heck you might not even need to create a new DB. Once you launch FF; it creates the standard *.sqlite DB in the new profile. All you have to do is read the sql statements back into their respective places.

    PS: Backing up the json files may not be a good option; because they are running in the past.

    On a side note; mozilla developer have changed the json format based on the newest EMCA code.

    “Firefox 3.5 dropped support for the JSON.jsm JavaScript module in favor of native JSON object support, as specified in the ECMAScript 3.1 draft specification”

    https://developer.mozilla.org/en/json#Using_JSON.jsm

    So, basically backing up json files is going to have draw backs because of changes to EMCA; unless you know how to parse and convert between formats. This is more than most people want to do for back ups.

    Otherwise just dump out the database and read the sql files back into the database. All of this can be done on the command line / scripted / etc.

    PS: Exporting the files via the GUI; you loose information like tags, etc. So, the switch to:
    sqlite
    json
    html

    formats has created a little issue for the users of Firefox.

    PSS: During the upgrade from HTML files to sqlite ; I lost tons of bookmarks. I had to parse the HTML file and re-add all my bookmarks (several thousand). Not sure if I got the parsing done correctly. But rebuild the folder structure and extracting the data took days; and weeks to manually re-add and sort them.

    I believe it was based on the default sqlite delimiter which is the “|” pipe. Several bookmark names had the pipe in the title which threw off sqlite database. The default delimiter should be something other than a “|” character. May a tab? Not sure how many site use a tab in their “title” tag.

    Translation; there was nothing that could be done about reimporting my bookmarks.

    FYI: I used the same bookmark file since the old netscape days. Yeah, it had some age to it.

    If the FF developer take a hint; the exporting of the bookmarks, cookies, etc should be done is sql; providing that they stay with that as a bookmark format. Exporting has HTML; well, they have to take into account things like tags and key words, etc.

    Just my .02 cents worth…. working on my own export facility so I don’t have to waste time doing parsing and worrying about upgrades.

    nix

    August 2, 2010 at 12:55 am

  5. Just a follow to my previous post; hopefully it helps someone.

    With respect to automating the export of bookmarks.

    1) Json files run in the past unless you use bookmark manager via the GUI to update the json files.

    2)Json files are in transition and who knows, if the file format will change again. Which will cause problems for users between upgrades.

    nix

    August 4, 2010 at 9:45 pm

  6. Thanks for this great entry! Just in case it’s useful to anyone, I’ve included my usage below. I added a line to get the most recently used profile directory to allow the script to be portable to my multiple computers. Of course this could be suboptimal in a troubleshooting situation…

    i also added a date stamp to the export file and cleaned up the other files.

    Thanks again!

    The contents of my bu_ffPlaces.sh
    #!/bin/bash

    # PURPOSE: backup firefox places
    # usually in preparation for clearing them from my cache

    # saved as ~/Dropbox/computer/scripts/bu_ffPlaces.sh
    # make executable with the command:
    # chmod +x ~/Dropbox/computer/scripts/bu_ffPlaces.sh

    # notes
    # i used ~ as a shortcut for the home dir
    # this won’t work unless you’re executing it as yourself
    # so you might want to expand it for cron and similar tools

    profDirName=`ls -t1 ~/Library/Application\ Support/Firefox/Profiles/ |head -1`

    cd ~/Library/Application\ Support/Firefox/Profiles/${profDirName}/

    cp places.sqlite places1.sqlite

    sqlite3 ‘places1.sqlite’ <<!
    .dump html
    .output moz_places.html
    select url,title from moz_places;
    !

    d=`d`
    ## d is the name of an executable script in my path
    ## containing the line:
    # echo -n `date "+%Y.%m.%d_%H.%M.%S"` | pbcopy ; echo `pbpaste`
    ## it returns a date stamp and copies the date stamp to the clipboard
    cp moz_places.html ~/archive/browser\ history/firefox_places_${d}.txt

    rm moz_places.html
    rm places1.sqlite

    echo "mozilla places backed up to ~/archive/browser\ history/firefox_places_${d}.txt"

    manducasexta

    October 20, 2010 at 10:55 pm

  7. […] Automating export of Firefox 3 bookmarks and history from command-line/cron September 20086 comments 3 […]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: