Yoshikoder

Batch Reporting

Will — Sat, 11 Sep 2010 09:40:44 +0000

If you want to run a Yoshikoder dictionary over a large number of documents more quickly than the Yoshikoder would do it, you can now use the new version of JFreq. Plus you get stemming, stopword removal and other preprocessing steps too, if you want them. Just drag your documents into the window, upload your saved dictionary (not project), select your preprocessing options, pick an output file, and press Run.

Horrid html

Will — Thu, 08 Jan 2009 10:26:52 +0000

First off, there’s an update to the YK Converter available from here. It’s a ‘related file’ for the Yoshikoder, for some reason. These updates are primarily because I’ve just finished working on a project that required scraping a lot of truly horrid web pages, and the current machinery wasn’t quite up to dealing with them. In fact, they bust everything except TagSoup.

So, now there’s a way to specify the encoding of the text or html page you’re working with, since we can’t always guess what that is. And also a cheeky way to grab a small section of text, which I’ve been thinking of as ‘Tuc’ because it deletes all the text that is not currently selected. It seemed just the thing for manually pulling out a snippet of text from a html-stripped page without a lot of editing effort.

Revisiting the conversion task pointed out to me that technology has moved on since I first wrote the code. In particular, the Apache POI project is now further along, and Tika, a general text scraping library is in the incubator. Ultimately I’d like the converter to be a thin but pretty wrapper around Tika. In any case I should update the POI components in the next release.

A little bit of infrastructure

Will — Mon, 05 Jan 2009 15:04:06 +0000

OK. About the transparency thing. There’s now a proper place to send bug reports and feature requests for the Yoshikoder. I had considered using the many options available from Sourceforge, but plumped for something that was much simpler, arguably more elegant, and most importantly: blue.

For those of you unfamiliar with all this sort of thing, here’s how it works:

You can ‘make a ticket’ in order to request a feature in a future version of the Yoshikoder or to report a bug you’ve found in a current one.

Tickets are not meant to be used as support requests (and I’ll delete them heartlessly if they are). For now you can email me directly for that.

A really good bug report ticket tells me exactly how to replicate the bug you found, step by step in a minimal example. A really good feature request ticket states in detail what your feature does and gives an idea about how it might look in the interface.

A final note about the terminology: Tickets are new when they’ve been proposed, open when I’ve started work on them, resolved when I think I’ve finished, on hold when I’ve put them on the back-burner, and invalid when I can’t replicate the bug or don’t want to implement the feature.

And another one

Will — Sat, 03 Jan 2009 17:14:53 +0000

Another preview (RC2) is available here. Most of the debugging happened on the Mac side, but it ought to go slightly better everywhere.

I expect that RC3 will turn up as the aftermath of a day of testing on Windows. That will probably have to wait a bit – XP running under Parallels from the other partition of my MacBook Pro is not really the best environment to see how the other half lives. If it that all looks ok, I’ll put it up on Sourceforge as version 0.7.

And since now’s the time to make resolutions, I’ve resolved to be a bit more transparent about development plans. Sourceforge tells me that the Yoshikoder is downloaded about 200 times per month, so I can no longer really reassure myself that it’s just me, some colleagues, and a handful of surprisingly patient email aquaintances that need to know what I’m up to.

Sneaky preview

Will — Thu, 01 Jan 2009 18:54:02 +0000

In a remarkably productive Christmas break I finally got working on the much neglected, at this point almost mythical ‘next version’ of the Yoshikoder. It’s not quite there yet, but I thought it might be nice to share a sneaky new year preview with you.

On the plus side, this version matches patterns with multiple words, links concordances to their parent documents, significantly streamlines the reporting, and exports stuff into more formats. On the minus side it’s still a bit buggy, and startlingly ugly on Windows despite my best efforts at imposing antialiasing and suchlike.

Also, I’ve removed scores from categories and patterns since I don’t think anyone uses them and I’m not sure they make sense. If you do use them, and would miss them, then now’s the time to yell.

When it’s all in better shape I’ll put it on Sourceforge and make a proper release.

In the meantime, it’s here. Happy new year.

Just let me know if you run into truly awful bugs…

RID dictionary corrections

Will — Fri, 19 Dec 2008 09:17:52 +0000

It seems the regressive imagery dictionaries for French and Portugese were not in good shape to import into the Yoshikoder.

Now they are.

Thanks to Sophie for pointing that out.

A small addition

Will — Sun, 11 May 2008 14:12:16 +0000

Folk have requested the possibility to run their dictionaries over all project documents at once to generate a ‘unified dictionary report’ mirroring the unified word frequency report. This function is now attached to a menu item in the latest preview release. It drops the resulting report straight into a CSV file, for easy import into whatever you like to do your data analysis in.

JFreq – a little command line tool

Will — Fri, 06 Jul 2007 11:19:36 +0000

Just a quick pointer to a bit of code that you might find useful, if you’re a command line kind of person.

JFreq is a simple word counter. It takes your text files, filters them various ways, and spits out a table of counts organized word by document. The handy bit is probably the filtering. JFreq can currently stem in 12 languages (courtesy of the lucene project). It can also remove currency references, number references, and stop words from a list you provide. Requires Java 5 or higher. Output is in UTF-8.

JFreq is already in use as the part of the backend of the Stata implementation of Wordscores, and has been used by the Wordfish folk for research on the EU. One of these days it will get a nice graphical interface, but given the speed of Yoshikoder development lately, that’s unlikely to happen soon. And now comes with a nice graphical interface if you don’t fancy the command line version.

Minor converter update

Will — Fri, 06 Jul 2007 10:41:47 +0000

There’s a tiny weeny little update to the converter, available from the usual spot. I’ve made the help a bit better, and it should feel a bit slightly more native on Windows.

Batch dictionary reports

Will — Mon, 05 Mar 2007 17:37:01 +0000

Folk have been asking about being able to run dictionary reports over all their documents rather than one at a time. Since the code for the next version of the Yoshikoder is in pieces around my bedroom, with several bits having rolled under the carpet or been borrowed by the cat, I’ve made a little program to do dictionary reports in a batch. This application currently lives here, and is wrapped up for Windows. Give it a project file from the Yoshikoder and it will run a dictionary report on every document in the project, and drop the results into a file. At least, that’s the idea.