The Accidental Rubyist

invalid byte sequence in UTF-8

TIL: ncurses cbreak and halfdelay

leave a comment »

I accidentally typed C-c quickly in a sample of rbcurse and found that the program crashed giving an error somewhere deep in logger. I repeated this in various programs, the line giving the error would be random. I tried various versions of ruby, wondered whether this had started happening after upgrading to Lion, or perhaps a bug in ffi-ncurses, or something with iTerm2, or maybe xterm-256color.

(I might add, that when i do a wgetch or getch in my app, there is "rescue Interrupt" clause to take care of this situation.)

After many hours, I finally found that if I press C-c slowly one after the other, my app takes it, but in quick succession, the 3rd or 4th causes the program to abort. After further research, i came down to the following:

1. using cbreak in ncurses causes successive Control-C’s to be taken as interrupts.
2. Using half-delay also puts ncurses back into cbreak mode.

I was using half-delay so that some complex key codes can be handled, and this includes using ESC in place of ALT/META. So pressing ESC followed by ‘w’ in quick succession is taken as ALT-w and not 2 key strokes.

Currently my working solution has been to comment off cbreak, and to use nodelay in place of halfdelay. It seems to be working as of now. No idea whether there is some key under some terminal that may fail. I have tested this out under many different TERMs and stuff. I hope this doesn’t make the application hog CPU.

Update: I cannot set nodelay in startup as this means that wherever in the application I might do a getch, it will be non-blocking and I will not get the key I am expecting. I found this while typing “gg” in a textview. So I have used a wtimeout and set it to 1000 for the meantime. Switching off nodelay by itself does not work for me since in my app, when the window gets an ESC it waits for the next character or ERR to see if its an Esc-alphabet.

Written by totalrecall

December 20, 2011 at 12:03 pm

Posted in ruby

Major restructuring in rbcurse 1.5.0

with 2 comments

I’ve just finished a massive operation of splitting rbcurse into 3 gems. rbcurse-core contains the core infrastructure and basic widgets. These should allow for most ncurses applications. I intend to focus on keeping this as well tested and stable, and backward-compatible as possible. I’ll also work on simplifying the code of core as much as possible. The earliest widgets have also made it into core since a lot of users are using them. Thus, menu and textarea have made it, even though the editable textarea will never be very stable, and its better to use Vim to edit bodies of text. Core does have some new work which is quite experimental but i hope to have it as an integral part of rbcurse and to stabilize soon.

I’ve moved some more complex widgets to rbcurse-extras. These are widgets you should require less often. I’ve moved Table to extras, the simple. non-editable version will be in core. Similarly, the editable version of listbox is in extras, but the readonly version will be in core.

Stuff that is usable but experimental, and not thoroughly tested is in rbcurse-experimental. Essentially, I really am leaving it to users to improve. Some of it could move to extras if it proves useful.

Henceforth, rbcurse will be a meta-gem that installs the other three. You may install only core and develop with that. The directory structures have also changed appreciably. I did this in order to make the movement into 3 separate repositories a bit easier.

BTW, the first part of this operation was separating code into code, extras and experimental directories. Then adjusting the ‘requires’ in each file. Shell and perl (pie) made this quite easy. The second part was to create separate repos. After reading many posts on stackoverflow, I managed to get that done. I hope the history of each file is correct.

I did all this in the 1.5.0 branch, not master, so pushing this to master was another small job. Again found some articles and help on SO. Then to create separate gems for thm repos and tie them together. Someone pointed me to rspec and I’ve used their model.

I still have all the examples lying in the old repo. These still need to be changed so they can run only using core or extras. Most of them use features from all or two gems. I still have to change the names of some widgets that have moved from core to extras or vice-versa (listbox and table), and replace messagebox and tabbedpane with the simpler rewrites. This should take me several days, esp rewriting the examples. I intend standardizing the core widgets in 1.5. I’ve dome work on stacks and flows for placing widgets which needs to be tested further and integrated fully. I’ve done work on color-formatting of text which needs to be integrated and the API firmed up.

1.5.x should mainly be simplifying and cleaning up stuff, removing cruft. Very few feature additions.

Written by totalrecall

November 17, 2011 at 11:58 pm

Posted in ruby

colorizing bash output on terminal (non-curses)

leave a comment »

I’ve found that my ruby command-line scripts (gems) that put information into sqlite tables and then report from there, take time to load. If not loaded for an hour, the command can take as much as 8-10 seconds to load. I’ve tried faster-rubygems but that has made no difference (I am on 1.9.2 on OSX Lion).

The load time is okay, when i wish to add some info, but when I keep checking, its not acceptable. However, my earlier bash scripts that do similar reporting (and addition) respond instantly. Now i am not going back to writing stuff in bash, since the syntax keeps befuddling me after all these years. However, it takes only a few minutes to report data from sqlite, and get instant responses. You can also throw in 256 colors to get a neat presentable output. Here are some things that can help you.

This is the meat of the script. Define a temporary input and output file

echo ".mode column" > $file
echo "select id, title, priority, status from mytable where status != 'closed' order by priority , id;" >> $file
sqlite3 bugzy.sqlite < $file >> $out

Now your output is in the outfile. You can less or most the file.

If you want to color the rows here are some options.
colorize is a neat program that allows you to define regexes that work on a line basis and color the entire line. There’s a switch for "only color the pattern not entire line", but you can’t have both. So I stuck with the default.

cat <<ZZZ > colorize.cfg
/ cancel/ black
/ closed / black
/ started / blue on_yellow
/ P1 / reverse
/ P2 / bold white on_blue
/ P3 / yellow
/ P4 / magenta
/ P5 / bold blue
colorize --config=./colorize.cfg $out | less

So here I create a config file for colorize to use. It does the rest. Notice it has attribute, background and foreground, although it often does not work if you give both attribute and other options. Also, it does not seem to support 256 colors.

If you are willing to write a few lines of sed, you can add a variety of other colors or backgrounds rather than the garish 8 or 16. I like shades of grey.

I’ve put the basic color codes into a file that can be sourced. I’ve added some for the gray scale also. You need 256 color support in your Terminal (OSX’s Terminal has it, so does iTerm2). gnu-screen will have to be recompiled with 256 (see previous post).

I found a function that lets you get any color.

 function EXT_COLOR () { echo -ne "33[38;5;$1m"; }

You can use it as "$(EXT_COLOR 128) hello". Or use backticks.

Now use these combinations in sed (warning: blog has eaten up slashes and maybe more).

text=$(sed "
  s/(.* started .*)/${BLUE}${ON_WHITE}1${DEFAULT}/g;

  s/(.* canceled .*)/${BLACK}1${DEFAULT}/g;



  " $out)
echo  "$text" | less

There, you have instant display of data from an sqlite3 table. No ruby load times.
There's also a project called ansi-color on google-code, however, you would use that if you are generating the output. Let me know if you come across any project that allows for using 256 colors in bash in a cleaner way.

Also, let me know if you know how to get rubygems to load faster. I am sure if wrote the above program as a stand-alone ruby program it would respond immediately.

Here's a sample screenshot output, it's still garish and yucky, but i did it at 2am. Will improve the colors soon.

Written by totalrecall

November 14, 2011 at 1:06 pm

Posted in ruby

OSX Lion compile screen with 256 colors

leave a comment »

Terminal now supports 256 colors. But screen is not compiled with 256, so no matter what you do in the screenrc file, you won’t get 256 colors. homebrew wont install screen since OSX provides it.

This blog gives you the steps to compile, and within a minute you’ll have a 256 color screen.

Written by totalrecall

November 11, 2011 at 7:40 pm

Posted in being_geeky, Mac OS X

Tagged with ,

Displaying colored documents / text in ncurses

leave a comment »

I have wanted to show documents or text with coloring (like syntax highlighting) for some time. Here is a quick take on this. The image below shows 2 textviews. One has text colored using the TMUX format that I liked. The next document is the output of dooby which uses ANSI escape sequences.

ANSI escape sequences don’t allow us to close a "span", all previous colors are disabled whenever a new color is specified. The tmux-format allows us to nest colors, so when a color ends, the previous one can continue.
The tmux format uses syntax such as : "#[fg=green, bg=yellow, bold]". I’ve added a "#[end]" to signify end of a span. I still have to work on substringing text with such markup but the basics are in place.

Any new markup format supplies a parse_format which yields either the text or the color and attributes. The rest is handled by rbcurse which currently converts the text to an intermediate format, an array containing text, color and attribute. This is used for printing colored text. Any further operations, such as finding the length of a line, or slicing the marked-up text happens in this internal format coded once.

Written by totalrecall

November 8, 2011 at 2:41 pm

Posted in ruby

Still unable to open multiple windows in tmux from tmux.conf

leave a comment »

Made my second attempt to move to tmux (from screen). No matter what I tried, and I checked several blogs and config files, but tmux only opens one window. All calls to neww or new-window are ignored.
However, once inside tmux if I give the "C-b c", or "tmux neww" command, then a new window is created. Even putting these commands in a shell-script does not work. The only way to get multiple windows was to create a shell script with each new-window command and call that after loading tmux.

Edit: I finally accidentally did find a shell script that does open multiple windows — seems it creates them detached and then attaches them. So, i’ve got tmux working but will make the switch a little later.

I am taking inspiration from tmux to allow users to put stuff on the rbcurse statusline using left and right. Also, i was studying tmux’s way of specifying colors for text, and may use that for the statusline and other text. This is still under consideration.

I’ve moved over to iTerm2 yesterday, Loved opening vim in 256 colors.

I am slowly moving over more ports to homebrew. Looks like the days of macports are over for me. I made a local recipe for "tal" (trailer alignment).

Written by totalrecall

October 31, 2011 at 8:38 pm

Posted in ruby

OSX Lion moved to msmtp from postfix

leave a comment »

Ever since upgrading to OSX Lion, postfix stopped working. So i finally installed msmtp which took a minute or two only. Configured sendmail to use it, and now all my scripts that use "mail" are able to send mail through msmtp.

However, mpack still does not work, and attachments look much better when sent through mpack. Unable to find anything on the net regarding configuring mpack for msmtp. uuencoding a file does not help as the file still appears as an attachment which has to be clicked. mpack does MIME encoding.

I tried using mpack to just encode the file "mpack -o out.html in.html" and then attaching the file, and sending it using mail or sendEmail, but the mail shows up as junk.

Are there any alternatives to mpack ? Any other way of MIME encoding a file and sending it using msmtp or any other client ?

Written by totalrecall

October 20, 2011 at 10:39 am

Posted in ruby