Thursday, January 27, 2011

Wolfram|Alpha using Factor

Wolfram|Alpha is a fun and useful "calculation engine" from the creators of Mathematica. Launched in mid-2009, it is available on the web, mobile devices, and from search engines. Last week, Wolfram announced version 2.0 of their API. As part of that announcement, they made it "free" for up to 2,000 non-commercial requests per month.

Since Factor has a visual REPL, I thought it would make a perfect client for Wolfram|Alpha. Starting today, you can do this:

The Wolfram|Alpha API website has both an API reference and an API explorer that can help you learn how it works. To get started, you need to sign up to receive an API ID.

USING: accessors formatting http http.client images.gif
images.http io kernel namespaces sequences splitting
urls.encoding xml xml.traversal ;

We will store the API ID in a symbol, that you can set:

SYMBOL: wolfram-api-id

The API is queried by creating URLs (from the query input and API ID) and parsing the response as XML.

: query ( query -- xml )
    url-encode wolfram-api-id get-global
    sprintf http-get nip string>xml ;

Inside the XML response are "pods" corresponding to the different types of information returned by Wolfram|Alpha. You can choose to receive the "pods" as plaintext, images, sounds, HTML, or Mathematica values. By default, the API returns plaintext and images.

We can create a wolfram-image. word that queries the API, extracts the "pods", outputs the title and images contained in each "pod":

: wolfram-image. ( query -- )
    query "pod" tags-named [
        [ "title" attr print ]
            "img" deep-tags-named
            [ "src" attr http-image. ] each
        ] bi
    ] each ;

Thats how to integrate with Wolfram|Alpha. Before using, remember to set your API ID:

( scratchpad ) "XXXXXX-XXXXXXXXXX" wolfram-api-id set-global

In addition to this, I made some minor style improvements (the gray titles and indentation in the screenshot), created a wolfram-text. word that prints the response as plaintext, and a wolfram. word that detects if you are running Factor on the command-line or the graphic interface and outputs text or images appropriately.

You can find the code on my Github.

Note: I also needed to make a few minor fixes to Factor's libraries to get this to work. They are not yet merged into the main Factor repository, but should be soon:

Monday, January 24, 2011

Github Vanity

There was a blog post on The Changelog yesterday describing a project that can give at-a-glance statistics about a Github user. The project is called vain and is open source. Since implementing APIs is fun (at least the first few times), I thought I would show how to implement the vain utility using Factor and the Github API.

USING: accessors assocs combinators formatting http.client
json.reader kernel math sequences sorting utils ;


We can experiment with the Github API in the Factor Listener, looking at seejohnrun, the creator of vain:

( scratchpad ) ""
               http-get nip json> .
            { "followers_count" 18 }
            { "login" "seejohnrun" }
            { "public_gist_count" 0 }
            { "public_repo_count" 23 }
            { "location" "Verona, NJ" }
            { "created_at" "2009/03/19 10:29:18 -0700" }
            { "type" "User" }
            { "id" 64965 }
            { "blog" "" }
            { "email" "" }
            { "name" "John Crepezzi" }
            { "company" "Patch" }
            { "permission" json-null }
            { "following_count" 11 }


We can use the set-slots word (similar to how we implemented Reddit "Top") to get the user details:

TUPLE: user blog company created_at email followers_count
following_count gravatar_id id location login name permission
public_gist_count public_repo_count type ;

: user-info ( login -- user )
    "" sprintf
    http-get nip json> "user" swap at
    user new [ set-slots ] keep ;

Similarly, we can access a list of public repositories for a specific user:

TUPLE: repository created_at description fork forks
has_downloads has_issues has_wiki homepage language name
open_issues organization owner private pushed_at size url
watchers ;

: repositories ( login -- seq )
    "" sprintf
    http-get nip json> "repositories" swap at
    [ repository new [ set-slots ] keep ] map ;

Using this, we have everything we need to implement vain:

: vain ( login -- )
        user-info {
            [ login>> ]
            [ followers_count>> ]
            [ public_repo_count>> ]
        } cleave
        "%s - %s followers - %s public repositories\n" printf
    ] [
        repositories [ watchers>> ] inv-sort-with [
                [ name>> ]
                [ watchers>> "%s watchers" sprintf ]
                [ forks>> "%s forks" sprintf ]
                [ fork>> "(FORK)" "" ? ]
            } cleave "%-25s %12s %12s %s\n" printf
        ] each
    ] bi ;

Try it

You can see the output for seejohnrun.

( scratchpad ) "seejohnrun" vain
seejohnrun - 18 followers - 23 public repositories
ice_cube                  255 watchers     15 forks 
database_validation        60 watchers      4 forks 
track_history              57 watchers      4 forks 
easy_translate             39 watchers      2 forks 
vain                       23 watchers      2 forks 
console_tweet              15 watchers      1 forks 
tweetStream4J               6 watchers      3 forks 
my_tunes                    3 watchers      1 forks 
locale_base                 2 watchers      1 forks 
Open-Stanza                 2 watchers      1 forks 
Pretty-Damn-Fancy           1 watchers      1 forks 
rstack                      1 watchers      1 forks 
dependable                  1 watchers      1 forks 
quick_short                 1 watchers      1 forks 
html_namespacing            1 watchers      0 forks (FORK)
usps                        1 watchers      0 forks (FORK)
dotfiles                    1 watchers      1 forks 
datejs                      1 watchers      0 forks (FORK)
redis-repeater              1 watchers      1 forks 
roflscale                   1 watchers      1 forks 
weatherbug                  1 watchers      1 forks 
gravatar_helper             1 watchers      1 forks 
columnizer                  1 watchers      1 forks


Some fun with Gravatar pictures:

The code for this is on my Github.

Saturday, January 22, 2011

Todo Lists

It seems the thing to do these days is to write "a better todo list". Probably there is at least one (maybe dozens) implemented in each programming language in existence. Factor even has its own todo web application.

When it comes to development, most developers keep lists of changes that need to be made or features that need to be implemented. Factor has its own todo list on the wiki. I know you are thinking what I'm thinking: wouldn't it be great if we could keep the todo list alongside the code? In any event, it would make a nice demonstration of the vocabulary and help browser system.


First, some background. Every vocabulary supports various metadata associated with the code, including:

summary.txta single line description of the vocabulary
authors.txta list of vocabulary authors
resources.txta list of files to include when deploying
tags.txta list of vocabulary tags used for organization
platforms.txta list of supported platforms if not cross-platform


We are going to add to this a todo.txt file containing a todo list of improvements or additions that could be made to the vocabulary. The format of the todo.txt file will be a list of text, each on its own line.

USING: arrays assocs formatting io io.pathnames kernel
sequences vocabs vocabs.loader vocabs.metadata ;

IN: todos

The path to the todo.txt file is relative to the directory containing the specified vocabulary:

: vocab-todo-path ( vocab -- string )
    vocab-dir "todo.txt" append-path ;

We can get and set the list of todo items using vocab-file-contents and set-vocab-file-contents, respectively.

: vocab-todo ( vocab -- todos )
    dup vocab-todo-path vocab-file-contents ;

: set-vocab-todo ( todos vocab -- )
    dup vocab-todo-path set-vocab-file-contents ;

We could add new todo items at runtime:

: add-vocab-todo ( todo vocab -- )
    [ vocab-todo swap suffix ] keep set-vocab-todo ;

Printing out the todo list for a specified vocabulary is pretty easy:

: todos. ( vocab -- )
    vocab-todo [ print ] each ;

Using the child-vocabs word, we can look through a vocabulary hierarchy for all todo files, returning a map of vocabulary to non-empty list of todo items.

: all-todos ( vocab -- assoc )
    child-vocabs [ dup vocab-todo 2array ] map
    [ second empty? not ] filter ;

And then print them out from the Listener:

: all-todos. ( vocab -- )
    all-todos [
        [ "%s:\n" printf ] [ [ "- %s\n" printf ] each ] bi*
    ] assoc-each ;

Try it

Although we could make the todo.txt files by hand, why not try using Factor?

( scratchpad ) USE: tools.scaffold

( scratchpad ) "foo" scaffold-work
Creating scaffolding for P" resource:work/foo/foo.factor"

( scratchpad ) "" scaffold-work
Creating scaffolding for P" resource:work/foo/bar/bar.factor"
Loading resource:work/foo/bar/bar.factor

( scratchpad ) "The first thing" "foo" add-vocab-todo

( scratchpad ) "The second thing" "foo" add-vocab-todo

( scratchpad ) "Another thing" "" add-vocab-todo

( scratchpad ) "foo" todos.
The first thing
The second thing

( scratchpad ) "foo" all-todos.
- The first thing
- The second thing
- Another thing

If you look in $FACTOR/work, you will now find the foo/todo.txt and foo/bar/todo.txt files that we just created.


We can use these words to make a dynamic help article containing all of the todo entries for loaded vocabularies:

USING: assocs help.markup help.syntax kernel todos ;

: $all-todos ( element -- )
    drop "" all-todos [
        [ $heading ] [ $list ] bi*
    ] assoc-each ;

ARTICLE: "vocab-todos" "Vocabulary todos"
{ $all-todos } ;

Once loaded, just run this to see the help article created by the previous example:

( scratchpad ) "vocab-todos" help

The code for this is on my Github.

Wednesday, January 19, 2011

Open URL

A useful library in Python is the webbrowser module: it allows you to open a URL in your web browser. When I was writing Reddit "Top", this was something I wanted but couldn't find in Factor. It would be great to have cross-platform "open URL" functionality, so I thought I would show how to build it.


Mac OS X comes with a command-line utility called open. It will open files or directories as if you had double-clicked them in the Finder. If you pass it a URL, it will open that URL in the default web browser. We can use the io.launcher vocabulary to run this command in a new process:

USING: formatting io.launcher urls.encoding ;

: open-url ( url -- )
    url-encode "open \"%s\"" sprintf try-process ;

Alternatively, you could use Applescript (via the osascript command) and the "open location" feature. This is how you might do it, if you want to target a specific browser ("tell application... activate OpenURL..."), indicate that it should open in a new window ("...toWindow..."), or open several URLs at the same time.

USING: formatting io.encodings.ascii io.launcher urls.encoding ;

: open-url ( url -- )
    "osascript" ascii [
        url-encode "open location \"%s\"" printf
    ] with-process-writer ;


On Linux, the situation is a little more complicated. Using Gnome, you can run the gnome-open command. Using KDE, you could run the kfmclient command. On other systems, maybe you could use mimeopen, or maybe write your own. For this example, we will assume you are running Gnome -- but you could support other methods and/or detect which method is appropriate to use:

USING: formatting io.launcher urls.encoding ;

: open-url ( url -- )
    url-encode "gnome-open \"%s\"" sprintf try-process ;


On Windows, we can use the ShellExecute function from Shell32.dll. In Factor, this is defined in the windows.shell32 vocabulary.

USING: urls.encoding windows.shell32 windows.user32 ;

: open-url ( url -- )
    url-encode [ f "open" ] dip f f SW_SHOWNORMAL ShellExecute drop ;

Try it

Once you have the appropriate open-url word loaded into your Factor VM, you should be able to try it out:

( scratchpad ) "" open-url

The code for this (designed as a cross-platform webbrowser vocabulary supporting open-url) is on my Github.

Note: It would be nice if clicking on URLs in the Factor browser and presentations would open them in your web browser, but I haven't yet figured out how to get that to work.

Monday, January 17, 2011

Reddit "Top"

Reddit has an API that can be used for accessing much of the information available through their website. We can retrieve a JSON list of recent stories posted to any subreddit by going to$NAME. You can experiment with this in the Factor listener - to retrieve top stories for the programming subreddit:

( scratchpad ) USING: http.client json.reader ;

( scratchpad ) ""
               http-get nip json> .

Someone once used the API to build a reddit-top program for monitoring top stories from the console. We will use Factor vocabularies to scrape Reddit and produce something similar:

We start by building a (subreddit) helper word to retrieve the JSON response for a particular subreddit, extracting the top stories, and returning an array of hashtables (one for each of the top stories).

: (subreddit) ( name -- seq )
    "" sprintf http-get nip
    json> { "data" "children" } [ swap at ] each
    [ "data" swap at ] map ;

We can then define a story tuple, with a slot for each attribute returned by the API.

TUPLE: story author clicked created created_utc domain downs
hidden id is_self levenshtein likes media media_embed name
num_comments over_18 permalink saved score selftext
selftext_html subreddit subreddit_id thumbnail title ups url ;

Once we have that, we can use the set-slots word from my previous post on setting attributes to build a subreddit word that retrieves the top stories as objects:

: subreddit ( name -- stories )
    (subreddit) [ story new [ set-slots ] keep ] map ;

Thats all we need to build the subreddit-top word demonstrated in the beginning:

  1. Retrieve the top stories for a given subreddit.
  2. Loop over each story.
  3. Format and print the relevant attributes.
: subreddit-top ( subreddit -- )
    subreddit [
        1 + "%2d. " printf {
            [ title>> ]
            [ url>> ]
            [ score>> ]
            [ num_comments>> ]
                created_utc>> unix-time>timestamp now swap time-
                duration>hours "%d hours ago" sprintf
            [ author>> ]
        } cleave
        "%s\n    %s\n    %d points, %d comments, posted %s by %s\n\n"
    ] each-index ;

This (and some code for users and comments) is available on my Github.

Friday, January 14, 2011

Setting Attributes

One test of dynamic languages is to try and set attribute values on an object dynamically (e.g., without knowing until runtime which attributes need to be set). Below, we compare a simple example in Python, a fairly dynamic language, to Factor:

class Foo(object):
    a, b, c = None, None, None

obj = Foo()

d = { "a" : 1, "b" : 2 }
obj.a = d.get("a")
obj.b = d.get("b")
obj.c = d.get("c")

print obj.a # 1
print obj.b # 2
print obj.c # None

We might directly translate the previous example to Factor code, using slot accessors to set attributes on the tuple instance:

TUPLE: foo a b c ;

foo new

H{ { "a" 1 } { "b" 2 } } {
    [ "a" swap at >>a ]
    [ "b" swap at >>b ] 
    [ "c" swap at >>c ] 
} cleave

[ a>> . ] [ b>> . ] [ c>> . ] tri

But, it's much better if you don't need to know ahead of time which attributes a class has (i.e., needing to write code to handle each attribute). In Python, you might instead set each value dynamically using the setattr function:

for name, value in d.items():
    setattr(obj, name, value)

We can use the set-slot-named word from the db.types vocabulary to do the same from Factor:

USING: assocs db.types fry kernel ;

: set-slots ( assoc obj -- )
    '[ swap _ set-slot-named ] assoc-each ;
Note: the set-slot-named word (and the offset-of-slot word that it uses) should probably be moved to the slots vocabulary.

We can simplify the previous example using our newly created set-slots word and try it in the Factor listener:

( scratchpad ) TUPLE: foo a b c ;

( scratchpad ) foo new

( scratchpad ) H{ { "a" 1 } { "b" 2 } } over set-slots .
T{ foo { a 1 } { b 2 } }

Thursday, January 13, 2011

Trashing Files: Part 3 (Windows)

In Part 1 and Part 2, we implemented send-to-trash on Mac OS and other Unix-like systems. In Part 3, we will be implementing support for the Windows Recycle Bin using Factor.

First, we create the vocabulary:

USING: accessors alien.c-types alien.strings
alien.syntax classes.struct classes.struct.packed destructors
kernel io.encodings.utf16n libc math sequences system trash
windows.types ;


We will be using the alien vocabulary to call the SHFileOperationW function from the shell32.dll library. Unfortunately, this function expects a "packed structure" (e.g., without data structure padding), so I needed to add support for this first (in the classes.struct.packed vocabulary). Using this, the PACKED-STRUCT: word creates a structure with each field aligned to a single byte.

LIBRARY: shell32


    { hwnd HWND }
    { wFunc UINT }
    { pFrom LPCWSTR* }
    { pTo LPCWSTR* }
    { fFlags FILEOP_FLAGS }
    { fAnyOperationsAborted BOOL }
    { hNameMappings LPVOID }
    { lpszProgressTitle LPCWSTR } ;

FUNCTION: int SHFileOperationW ( SHFILEOPSTRUCTW* lpFileOp ) ;



With these defined, we can implement send-to-trash, by simply creating the SHFILEOPSTRUCTW structure (making sure to add extra null bytes to the end of the path being trashed -- since it should be "double null terminated"), and then performing the SHFileOperationW function.

M: windows send-to-trash ( path -- )
        utf16n string>alien B{ 0 0 } append
        malloc-byte-array &free

        SHFILEOPSTRUCTW <struct>
            f >>hwnd
            FO_DELETE >>wFunc
            swap >>pFrom
            f >>pTo
            FOF_NOCONFIRMATION bitor
            FOF_NOERRORUI bitor
            FOF_SILENT bitor >>fFlags

        SHFileOperationW [ throw ] unless-zero

    ] with-destructors ;

The code for this is on my Github.

Wednesday, January 12, 2011

Trashing Files: Part 2 (Unix)

In Part 1, we implemented send-to-trash on Mac OS. In Part 2, we will be adding Factor support for the Trash Specification used on other Unix systems (e.g., Linux or BSD).


First, we need to create the trash.unix vocabulary:

USING: accessors calendar combinators.short-circuit environment
formatting io io.directories io.encodings.utf8 io.files io.files.types io.pathnames
kernel math math.parser sequences system trash unix.stat
unix.users ;

IN: trash.unix

When trashing a file, we sometimes need to look for the "top directory" of a mounted resource that contains a given path. We can use the lstat function (using the link-status word from unix.stat) to read information about the file or symbol link pointed to by a path. If the file system details are different between a path and its parent directory, then it is the top directory of a mounted resource.

: top-directory? ( path -- ? )
    dup ".." append-path [ link-status ] bi@
    [ [ st_dev>> ] bi@ = not ] [ [ st_ino>> ] bi@ = ] 2bi or ;

: top-directory ( path -- path' )
    [ dup top-directory? not ] [ ".." append-path ] while ;

We need to be able to create trash directories with "user-only" permissions:

: make-user-directory ( path -- )
    [ make-directories ] [ OCT: 700 set-file-permissions ] bi ;

To be a valid trash path, we need to check:

  1. The path is to a directory
  2. The path has the sticky-bit set
  3. The path should not be a symbolic link
: check-trash-path ( path -- )
        [ file-info directory? ]
        [ sticky? ]
        [ link-info type>> +symbolic-link+ = not ]
    } 1&& [ "invalid trash path" throw ] unless ;

The Trash Specification defines various locations for the trash directory, in order of preference:

  1. In $XDG_DATA_HOME/Trash (or $HOME/.local/share/Trash), if the file being trashed is on the same mount point.
  2. In the top directory of the path's mount point, $TOPDIR/.Trash/$UID, if the .Trash directory is available.
  3. In the top directory of the path's mount point, $TOPDIR/.Trash-$UID, in a user-created directory.
: trash-home ( -- path )
    "XDG_DATA_HOME" os-env
    home ".local/share" append-path or
    "Trash" append-path dup check-trash-path ;

: trash-1 ( root -- path )
    ".Trash" append-path dup check-trash-path
    real-user-id number>string append-path ;

: trash-2 ( root -- path )
    real-user-id ".Trash-%d" sprintf append-path ;

: trash-path ( path -- path' )
    top-directory dup trash-home top-directory = [
        drop trash-home
    ] [
        dup ".Trash" append-path exists?
        [ trash-1 ] [ trash-2 ] if
        [ make-user-directory ] keep
    ] if ;

We need to implement some logic to handle name collisions (e.g., when trashing a file with the same name as a file already in the trash directory). To do this, we use "safe" filenames (adding an incrementing extension to ensure uniqueness):

: (safe-file-name) ( path counter -- path' )
        [ parent-directory ]
        [ file-stem ]
        [ file-extension dup [ "." prepend ] when ] tri
    ] dip swap "%s%s %s%s" sprintf ;

: safe-file-name ( path -- path' )
    dup 0 [ over exists? ] [
        [ parent-directory to-directory ] [ 1 + ] bi*
        [ (safe-file-name) ] keep
    ] while drop nip ;

And, finally, we can implement the send-to-trash logic:

  1. Lookup the trash path for the file being trashed
  2. Move the trashed file into a files sub-directory, using a safe file name
  3. Create an "information file" in an info sub-directory, with details of the trashed file.
M: unix send-to-trash ( path -- )
    dup trash-path [
        "files" append-path [ make-user-directory ] keep
        to-directory safe-file-name
    ] [
        "info" append-path [ make-user-directory ] keep
        to-directory ".trashinfo" append [ over ] dip utf8 [
            "[Trash Info]" write nl
            "Path=" write write nl
            "DeletionDate=" write
            now "%Y-%m-%dT%H:%M:%S" strftime write nl
        ] with-file-writer
    ] bi move-file ;

The code for this is on my Github.

Monday, January 10, 2011

Trashing Files: Part 1 (Mac OS)

Most operating systems provide support for sending files to the "trash can" (or sometimes "recycle bin"). Inspired by a python project called "send2trash", I thought Factor should have a similar cross-platform library for trashing files.


First, we are going to define a trash vocabulary, and use a HOOK: that dispatches to the proper implementation, depending on which operating system you are running.

USING: combinators system vocabs.loader ;

IN: trash

HOOK: send-to-trash os ( path -- )

    { [ os macosx? ] [ "trash.macosx"  ] }
    { [ os unix?   ] [ "trash.unix"    ] }
    { [ os winnt?  ] [ "" ] }
} cond require


Next, we will create the trash.macosx vocabulary.

USING: alien.c-types alien.strings alien.syntax classes.struct
core-foundation io.encodings.utf8 kernel system trash ;

IN: trash.macosx

On the Mac OS, there are several methods of moving files to the trash. A good discussion on CocoaDev lists some of them. We are going to use the alien vocabulary to make calls into the File Manager in the CarbonCore.framework. Some functions will return an OSStatus flag (a signed 32-bit integer) to indicate if the operation succeeded. We will add a TYPEDEF: for it, and then define the GetMacOSStatusCommentString function that converts the status flag into a human readable error.

TYPEDEF: SInt32 OSStatus

FUNCTION: char* GetMacOSStatusCommentString ( OSStatus err ) ;

: check-err ( err -- )
    [ GetMacOSStatusCommentString utf8 alien>string throw ] 
    unless-zero ;

Many of the file operations act on an FSRef structure which represents a path within the file system. We will define the FSPathMakeRefWithOptions function which will allow us to create these references:

STRUCT: FSRef { hidden UInt8[80] } ;

TYPEDEF: UInt32 OptionBits

FUNCTION: OSStatus FSPathMakeRefWithOptions (
    UInt8* path,
    OptionBits options,
    FSRef* ref,
    Boolean* isDirectory
) ;

We can then make a <fs-ref> word for creating references, given a path to a file (or directory).

CONSTANT: kFSPathMakeRefDoNotFollowLeafSymlink HEX: 01

: <fs-ref> ( path -- fs-ref )
    utf8 string>alien
    FSRef <struct>
    [ f FSPathMakeRefWithOptions check-err ] keep ;

There are several ways of "trashing" files, but one recommended way is implemented by the FSMoveObjectToTrashSync function:

FUNCTION: OSStatus FSMoveObjectToTrashSync (
    FSRef* source,
    FSRef* target,
    OptionBits options
) ;

Implementing the send-to-trash word is now pretty straightforward:

CONSTANT: kFSFileOperationDefaultOptions HEX: 00

M: macosx send-to-trash ( path -- )
    <fs-ref> f kFSFileOperationDefaultOptions
    FSMoveObjectToTrashSync check-err ;

You can test this by creating a temporary file (e.g., /tmp/foo), sending it to the trash, and then verifying that it exists by looking in the Finder's Trash.

( scratchpad ) USING: trash io.encodings.ascii io.files ;

( scratchpad ) "" "/tmp/foo" ascii set-file-contents

( scratchpad ) "/tmp/foo" send-to-trash

Note: This method does not appear to support the "Put Back" functionality (to "undo" the trash operation). Perhaps there is some metadata that we can add (or a different function we can call) that will track the original file location so that the Finder knows where it should be restored to.

The code for this is on my Github.