Songcache Modification Tool
- This topic has 33 replies, 4 voices, and was last updated 2 years, 1 month ago by
.
-
Posts
-
RB3 Songcache File Scripty Modification Tool Jobber, aka ‘CacheTools’
This utility allows you to write custom scripts to modify your RB3 ‘songcache’ file. Typically this would be to accomplish various metadata corrections you might wish to make on a ‘batch’ basis.
It’s also possible to do stranger things with it, but I’ll leave that up to your imagination for now.
Latest Release: v0.9 alpha (February 24, 2020)
Download Link (Now dead!)
This is still alpha software!
Good luck to all who take up the challenge. Hopefully no horrible bugs crept in during my rush to release it. Minimal documentation is found in the following two posts, which are pasted from text files in the zip archive.
Have fun poking at it and please tell me what you think, both how it works now and what it might become in the future.
(Screenshot of the exhilarating and groundbreaking GUI…)
20200224
v0.9 alpha
RB3 Songcache File Scripty Modification Tool Jobber, aka ‘CacheTools’
This utility allows you to write custom scripts to modify your RB3 ‘songcache’ file. Typically this would be to accomplish various metadata corrections you might wish to make on a ‘batch’ basis.
It’s also possible to do stranger things with it, but I’ll leave that up to your imagination for now.
But Before You Continue…
WARNING: I’ve confirmed that if the rewritten cache exceeds 0xAA (170) blocks in size (perhaps in the 1.4MB range on-disk), it will be considered “read-only” by RB3. This means it will pretend to rebild your cache EVERY TIME with subsequent additions, though it will keep your old cache tool changes intact.
(This threshold is documented to cause another tree level to be used in the hash table, but I’m not handling it capably (apparently). Most likely I’m not passing the right parameters to the X360 library API and it’s just assuming what it thinks are “sane” defaults.)
However, for a smaller library your songcache would still be fine (i.e. “rewritable”) and will expand when adding new customs (without rescanning). But with a billion customs at our disposal, who has a library that small? I suppose the trick would be to start from a “seed” cache (for example, right after an initial RBHP build), run all your scripts, and then build from there.
That said, if anyone can send a knowledgeable X360 or STFS coder my way to help me out, I’d appreciate it!
Requirements
I run on Windows 7, so that’s all I can guarantee it’ll work on.
You’ll need some sort of baseline .NET Framework to run this under. At the moment I’m not particularly motivated to figure out which definitive version, but hopefully you can figure that out for yourself. If you can’t, then running alpha software probably isn’t your bag, and I’d be loathe to have to teach you the particulars of writing scripts.
Also, I only have an Xbox 360, so those are the sorts of cache files I deal with. I build to a CON file that should load onto a USB stick with no issues. If you’re on another console, you could still use Nemo’s C3CONTools to extract the cache from the CON and do what you will with it.
Installation
Extract the zip archive into a directory. That’s pretty much all you need.
Usage
0) Added a step zero as a reminder: Always make a backup of your cache before overwriting it!
1) Load a songcache file. You should be able to select either a full CON file or an extracted songcache payload. As of v0.9, hold down Shift key before cache loads to see lots of info (SLOW-ASS ON BIG CACHES!).
2) Run a script file. It should tell you if it loaded and/or executed successfully.
2b) If you’d like, run another script file. You can do this as many times as you need, really, but remember that the results are cumulative: if you need a ‘reset’, just go back to Step 1 and reload the script.
3) Save the modified songcache file. Keep in mind, it should be named ‘songcache’ by the time you put it on a USB stick to use on the 360.
How to Write Scripts
That’s in another document.
How to Get Help
For now, there’s a thread on C3 where you can petition the community at large:
http://customscreators.com/index.php?/topic/15260-songcache-modification-tool/
Legal Crap
USE AT YOUR OWN RISK. I’m not responsible for the trouble you will inevitably cause yourself by mucking about this program. Always make a backup of your cache before overwriting it!
Do not host this program on your site or any other public archive.
Do not “deep” link the file download directly, as it removes any context for people to get help.
Instead, link to the C3 announcement thread:
http://customscreators.com/index.php?/topic/15260-songcache-modification-tool/
Do not sell this program in any form, including bundles or archives, virtual or physical.
Otherwise, have fun! It’s free!
Credits
My name is Markleford Friedman, and I take full responsibility for this utility when the jackboots come!
That said, thanks to some folks at C3 are in order: TrojanNemo for sample code and the patched X360 library, pksage for the nascent songcache file format document, and Farottone for being Farottone!
” src=”/wp-content/uploads/invision_emoticons/default_SA_smile.gif” />– m
History
20200224 v0.9 alpha
– Added “import”, “export”, “macro” (Thanks to hentaiphd for idea and collab)
– Added “verbose load” mode: hold down Shift key before cache loads to see lots of info (SLOW-ASS ON BIG CACHES!)
20200209 v0.8 alpha
– Fixed weird bug with Bob Marley RB3 pack (aka “male_premium_hendrix_woodstock”): thanks FujiSkunk!
20190321 v0.7 alpha
– Tool now automatically fixes “missing” album art on songs with intact graphics (Promised Land, Charlene, Ghostbusters, Accidentally in Love)
20181231 v0.6 alpha
– Version number label on UI for easy identification
– Error message cleanup
– Added support for trackNumber (“track” “num” “number” “tracknum” “tracknumber”)
– Additional song attribute aliases: see SCRIPTING-HOWTO.txt
20161025 v0.5 alpha
– First public release
[end]
20200224
v0.9 alpha
SCRIPTING-HOWTO
DESIGN PHILOSOPHY
I had my own needs for fixing song metadata in the cache, but I figured making something that would satisfy all comers might be cool.
But hey, for all I know nobody else is gonna care to use it! So I may as well have some fun toward my ends, and that included taking on a programming challenge while writing novels and producing albums for the past few years. Gotta keep the those skills up!
The project direction pretty much came down to the choice of “Script-like or SQL-like”? While the songcache could certainly be seen as a sort of database to run structured queries against, I eventually figured that writing a scripting grammar would provide better experience to leverage in future projects.
So a scripting system it is. Simple for now: no symbols or extended expressions, no user-declared variables, and a limited number of attributes to get and set.
GRAMMAR
Okay, so I forgot the right notation for a proper BNF (comp-sci degree almost three decades old now!), but this is what I came up with when designing the script grammar…
SCRIPT :: BLOCK
BLOCK :: LINE *
LINE ::[ REM ]
BLANK :: (WHITESPACE)
REM :: ; (TEXT)
STRING :: "(TEXT)"
ATTRIBUTE** :: song[name] | id | shortname | artist | album | yearrecorded | yearreleased | vocalparts | gameorigin | genre | rating | gender
COMMAND :: set ATTRIBUTE to EXPRESSION | using ATTRIBUTE replace STRING with EXPRESSION | print[ln] EXPRESSION | import| export | macro
SELECT :: each SELECTOR do BLOCK end
SELECTOR :: ATTRIBUTE within SET
SET ::*
BRANCH ::CONDITION BLOCK [elsif CONDITION then BLOCK] * [otherwise BLOCK] end
CONDITION :: CLAUSE [ and CLAUSE | or CLAUSE ] *
CLAUSE :: COMPARATOR | SELECTOR
COMPARATOR :: ATTIRIBUTE [eq | ne | contains] EXPRESSION
EXPRESSION ::[ + EXPRESSION ]
** NOTE: Attribute aliases have expanded after first release, since even I’ve forgotten what I used for the official names for them! These days I’m supporting this list:
“song” “name” “songname”
“id”
“shortname”
“artist” “band”
“album” “title” “albumtitle”
“track” “num” “number” “tracknum” “tracknumber”
“year” “released” “yearreleased”
“recorded” “yearrecorded”
“vocalparts” “vocals” “parts”
“gameorigin” “game” “origin”
“genre”
“rating”
“gender”
** Whether supporting all these alternates is wise in the long run, I’m not quite sure!
MACROS
As of v0.9 I’ve added support for special-purpose “macros”. This is the “Wild West” of in-development functionality: error checking isn’t at all robust, so you might get a crash if your parameters aren’t just right! This is going to need a lot of testing.
** Package **
The first will add or remove songs from packages.
macro “package
“ An example add:
macro “package blackholesun 3”
To remove, make the songId negative:
macro “package blackholesun -3”
QUIRKS and CAVEATS
Capitalization of keywords doesn’t matter.
To print a quotation mark in a string, escape it with a backslash: “Here are some “quotes”.”
For now, there are no proper expressions. They’re merely string concatentations. (In fact you currently don’t need a ‘+’ to connect them, but I imagine you’ll want to use them in case that changes in the future.)
To that end, integers are compared as strings. Everything is stringified for comparing! When you want to set an attribute with a numeric value, though, I attempt to parse back to integer: if that fails, then no value is set.
Furthermore, there is no range/value validation for integer attributes. Set them at your own risk.
Chaining if/branch conditions with ‘and’/’or’ is pretty much only meaningful if you don’t mix conjunctions, since we currently don’t have expressions advanced enough to allow parentheses for ‘order of operation’.
EXECUTION, or HOW THESE THINGS ACTUALLY WORK
The program iterates over every song found in the songcache. Note that this includes any song that your console ever saw since the last full cache build, so there could be some crap customs still in there that you deleted ASAP after testing them (aherm, I’m looking at you, NON-PITCHED VOCAL SONGS!).
For each song, it runs the selected script in its entirety, top to bottom. During this execution, the named attributes only apply to the CURRENT song scoped by the iterator. (NOTE: “current” scoped song changes if you “import” a binary song chunk that does not match the current song. This allows you to import several at a time in a “once” block.)
This, however, means there could be a bit of wasted CPU cycles in comparing the same conditions over and over (especially string matching). Not that it takes more than a fraction of a second to run a script on a songcache of moderate size (unless you use print/ln a lot: that is quite costly!), but you can use the ‘once’ and ‘each’ branch types to help with this: after their conditions pass as ‘true’, they’re short-circuited to always returning ‘false’ and are never checked again.
FINAL ADVICE
ALWAYS keep a backup of your cache before poking at it!
You WILL mess it up at some point and will want to roll back!
EXAMPLES
In my experience, having a half-assed BNF to look at is nice, but you really learn the most from examples. Take a look at the included scripts.
Also, here are some very basic snippets to learn from:
; 'Comments' start with a semicolon
; They can be on lines by themselves
; Or a comment can follow a line
; Everything on a line after the semicolon is ignored
println song ; Printing out song name
; Strings are in double quotes
if name eq "Proud Mary" then
set vocalParts to 3
end
; Create an expression at any time by concatenating strings
println "Parsing song " + id + " (" + shortname + ")"
; Ids are numeric
if id eq 1006117 then
set vocalParts to 3
end
; contains = search within a string
; Label all songs charted by Farottone
if shortname contains "far_" then
set name to name + " (FAR!)"
end
; For now, we allow 'contains' with numbers
; Here, non wipe-proof song IDs always start with 10746
if id contains 10746 then
println name + " not wipe-proof"
end
; Nested ifs, of course
if artist eq "David Bowie" then
if album eq "Lets Dance" then
set album to "Let's Dance"
end
end
; Chaining if conditions, doing the same thing
if artist eq "David Bowie"
and album eq "Lets Dance" then
set album to "Let's Dance"
end
; The OR case
if artist eq "Queensryche"
or artist eq "Queensreich" then
set artist to "Queensrÿche"
end
; 'once' = like an if, but after the first time it matches we never look for it again (efficiency)
once shortname eq "proudmary2" do
set vocalParts to 3
end
; An extended chain:
; Use as name elsif's as needed,
; End with an 'otherwise' as catch-all case
if artist contains "Creedence" then
if name eq "Proud Mary" then
set name to "Proud Mary" + " (with Harmonies)"
elsif name eq "Fortunate Son (Original Version)" then
set name to "Fortunate Son (Not a Lame Cover)"
otherwise
set name to name + " (CCR roolz!)"
end
end
; Matching a string, then replacing it
if name contains " (Original Version)" then
using name replace " (Original Version)" with "" ; Use the null string ("") to delete substr
end
; Matching a set
if artist within "Muse" "Rush" "David Bowie" then
; Prefix released year on these albums to be able to sort chronologically
set album to released + ". " + album
end
; 'each' = like matching a set, but only once for each: when we find it, remove from list
; Once every member is found, we skip the entire check
each shortname within
"badmoonrising" "fortunateson2" "downonthecorner" "whollstoptherain" "proudmary2"
do
set artist to "Creedence Clearwater Revival (CCR)"
end
; Export "Black Hole Sun" from a TU5 cache
once shortname eq "blackholesun" do
export id + ".song"
end
; Import "Black Hole Sun" into TU5 from an export
once id eq id do
import "3.song"
macro "package blackholesun " + id
end
[end of SCRIPTING-HOWTO]
Just saw you updated the posts. Read through it all, good job. Sounds very cool. You need a screenshot in the OP
BAM!
BAM, indeed. Attached to OP.
Thanks for the screenshot: now people are sure to want to use it!
” src=”/wp-content/uploads/invision_emoticons/default_SA_wink.gif” />It looks nice and seems to work fine on the surface. I unfortunately don’t have time to actually put it to the test, but i’m sure others will. I’m glad I was able to help get you going on the right direction.
Awesome program! Nice job.
Amazing piece of software man.
” src=”/wp-content/uploads/invision_emoticons/default_SA_smile.gif” />Thanks, peoples!
” src=”/wp-content/uploads/invision_emoticons/default_SA_smile.gif” />But answer me this: Is the concept at all useful? It certainly makes for a fun little curiosity, but given that people can already make changes in the dta (made even easier by Nemo’s Quick Editor), even I come up short for identifying a ‘big win’ use case for doing it programmatically, particularly since the changes get wiped out if your cache is rebuilt.
The closest I’ve come to “uniquely powerful” is a yet-unreleased system to consume a .csv file of song reviews from both the missus and me that turns them into a cross-referenced series of custom genres. In this manner, it’s quite easy for us to filter all of the songs that we’ve ranked 4 or 5 lighters, which makes the ‘party shuffle’ feature actually useful for a change.
Also along those lines, I use those .csv reviews to make it look like the songs that both of us have rated 2 or 1 have Pro Guitar support, which means all we have to do is tick the ‘Pro Guitar NO’ filter and instantly all the crap from exports is filtered out, yet is still available if we have party guests that like the crap.
” src=”/wp-content/uploads/invision_emoticons/default_SA_wink.gif” /> (Mind, it works for us because we don’t have a Pro Guitar.)Both of those experiments, though, have too many moving parts for public release at this point.
So part of the reason for releasing the scripting system to the world is to see if other people can make something useful of it. To that end, if you manage to come up with something that’s useful to you (beyond what you can do with Quick DTA Editor), please share your little victories!
” src=”/wp-content/uploads/invision_emoticons/default_SA_smile.gif” />I’m gonna be honest and say I’m not gonna use it because it’s not what I would usually do. However, the same can be said for a particle collider, it’s still amazing even if I don’t use it.
” src=”/wp-content/uploads/invision_emoticons/default_SA_smile.gif”>It’s definitely cool and has great potential…but for me at least, it’s 2+ years too late. I haven’t fired up my Xbox 360 in, what, a year? If I ever play RB3 again I certainly won’t mess around with things. Unfortunately for those of us who have created tools for this community (now that includes you), there’s little activity going on and there’s no guarantee that anything you put out will get a ton of use. It is likely, however, that one or two people will use it and think it’s the best thing in the world. But those will be limited cases. For me, for example, it’s a mystery that cPlayer isn’t much more popular. I think it’s so great at what it does. But it also probably came out too late.
Yeah, even if I managed to launch this before the tail-end of the scene, it’s an “esoteric” idea to begin with! That’s why I need help in figuring out whether it has a niche that I can’t quite see yet, since even I’m coming up short for practical uses for it. I wasn’t thinking “practical utility” when I decided to do a scripting engine, just “challenging exercise”: my particle collider, if you will…
” src=”/wp-content/uploads/invision_emoticons/default_SA_wink.gif” />Oh, I forgot one other thing that I use it for, which was actually included as a script example:
if artist within "Muse" "Rush" "David Bowie" then
; Prefix released year on these albums to be able to sort chronologically
set album to released + ". " + album
end
I hate the way RB3 sorts albums alphabetically, so this was my fix for bands with a lot of customs. It’s actually something that it’s quite suited for, rather than going into all the DTAs and fixing by hand.
Actually, most of my cache experiments have been of the sort of “HMX really dropped the ball on filtering and sorting, let’s see how we can work around it”. Seriously, why didn’t they pay more attention to allowing players to “curate” their collections? Why in the world would they stop at the TU5 feature of “don’t show 1-lighter songs” instead of providing an actual filter for “review”? Why do playlists seem so tack-on?
The mind boggles, at times, why such low-hanging fruit slips through their fingers, and from what I’ve seen from RB4 they’ve yet to improve on library management much at all.
Been away from here forever, but I got a message that the KeepItFishy link is down.
I’ve now put it up for download here also:
- You must be logged in to reply to this topic.