[espher]

pksage note: A decision has been reached on some, but not all, of the following topics. For more information, see this post in this thread.

 

We’ve been having some on-again, off-again discussions regarding the playtesting and track submission process, and I would like to present a preliminary draft of a new process for your consideration and review, as you are stakeholders in this process. As I’m sure many of you have noticed, there have been several issues (of varying severity) with track submissions, scheduling, playtesting and release coming from author, playtester, and administrator perspectives, and I would like to inject a little more “bureaucracy” into the process (but not too much) to ensure songs are getting the care they deserve and that releases flow as smoothly as possible.

 

The key issues that I have, personally, identified with the existing process are:

 

a) Songs are being scheduled and entering playtest before they are close to ready.

We, as administrators, have not been sufficiently active in filtering/screening songs ahead of time.

c) Playtesting is minimal or non-existent.

d) Every Monday through Wednesday is panic mode central every week.

 

In broad brushstrokes, here is the process I would like to implement for future song scheduling, testing, and release.

 

Note that this change in process will likely come with a full purge of the playtesting group, with a new round of recruitment. Existing playtesters are more than welcome back but they will need to adhere to the new testing process.

 

Scheduling & Release

• Songs must be scheduled AND the song must enter testing at least three full weeks prior to the scheduled release date. Three weeks seemed to be the “fairest” number to ensure that the song gets sufficient review and testing, and it gives us flexibility with respect to song shuffling. Authors with a proven track record and/or releases trying to fit a specific event/theme (e.g. holiday releases, releases to coincide with a greatest hits album, anniversary of xyz events) may be granted exceptions, but these will be evaluated on a case-by-case basis.
  • Authors can pick any release date that they choose, but the preference will be to fill holes in the schedule before adding on to an already full week. A week with a “hole” is something that we have loosely defined as any week with fewer than three songs, so long as one of those songs is a “big” release/hit, or any week with fewer than six songs for B-Sides/smaller artists/niche genres.
  • If you schedule a song a month or two out, and we hit three weeks prior to release and there is nothing ready for testing, the song will be removed from the schedule.

  [*] The current status of songs will be reviewed a week out by members of the admin team.

  • If the song is in good shape at that point, it will be “cleared” for release, with the expectation that a final version for video recording (can change slightly) is uploaded and available by the Monday prior to release, with the final “for release” version of the track uploaded and available by the Thursday prior to release (giving time to make minor fixes). This ensures that our video guy (currently Nemo) doesn’t have to rush through video prep, and that we’re not waiting with bated breath for a final upload.
  • If the song is not in good shape, it will be bumped (if it is part of a pack, the entire pack will be bumped). We will set a new tentative date (though the author can see about putting it somewhere else). If the song was insufficiently tested, I’ll deal with that problem. An author repeatedly having songs bumped will be contacted so that we explore a better workflow for all involved.
  • If we end up in a situation where many songs are bumped from a given week, and nothing can be brought forward or pulled from reserves to fill the hole in the schedule, we might have a Panic Mode week where we get people to roll up their sleeves and fix charts. Given the process, though, we should never run into this hurdle.

 

Playtesting

• As mentioned under scheduling & release, songs must enter testing at least three full weeks prior to the scheduled release date.
• The first playtest version must, at a minimum, contain Expert difficulty charts for all instruments present in the song.
• When songs are submitted for testing, they will be first given a “preliminary review” by a member of the admin team.
  • The preliminary review is mainly to identify if there are any easily visible issues with the chart. If they are visible but minor, they will be highlighted in general strokes for the author or a playtester to review for specific feedback. If they are visible and major — such as a chart that requires a rewrite, or parts that are missing altogether — the author will be advised of such and the song will be “pulled” from the testing queue (e.g. it’ll be unscheduled, and I’ll probably lock the thread) until a revised version is submitted and clears this preliminary review.

  [*] When songs pass preliminary review, they will be opened up for playtesting.

  • The plan at this time is to bring in structured playtesting, where I assign songs to a pool of playtesters based on their strengths, while giving them the flexibility to do extra songs if they want — the current laissez-faire style leads to a number of songs not getting tested or testers not even giving feedback/doing tests. The nearest releases will be given priority over future releases for testing purposes. Playtesters WILL be held accountable for the work they do (or don’t do), both in volume and in quality of feedback.
  • Authors will be expected to keep on eye on the songs of others and pitch a hand in for testing when something gets missed or isn’t getting any attention, though this will not be compulsory. That being said, while I am rebuilding the playtester pool, you may be asked to review/test songs — on the instrument you feel most confident authoring — to ensure no lemons are released.

  [*] A week out from the scheduled release, the song will be given another review pass to see how it holds up and see how feedback was implemented/addressed, and a release decision will be made as per the scheduling & release section above.

 

My hope is that the implementation of these changes will address the issues identified above as follows:

 

a) Songs are being scheduled and entering playtest before they are close to ready.

Requiring that songs are submitted three weeks out ensures that we have sufficient time to give the tracks a quick overview, and it also prevents someone from sliding a song in a week and a half our and disrupting review/testing flow. The preliminary review will ensure that songs that need a ton of work do not pre-occupy or eat up test time from playtesters or other authors. These changes, along with some other potential changes down the road, will also avoid the “new author whose capabilities we don’t really know is submitting a 12-pack of his favourite songs for next week” situation which could be disastrous (either being a big artist that gets more scrutiny or an unknown artist that nobody tests).

 

We, as administrators, have not been sufficiently active in filtering/screening songs ahead of time.

Implementing a two-stage release screening process puts some structure into that and ensures that we are at least signing off on and accountable for what does go out the door.

 

c) Playtesting is minimal or non-existent.

Structured playtesting ensures every song gets a look — and if they don’t, we can kick out testers that are slacking. The preliminary review ensures that authors are sufficiently reviewing their own stuff, either before submitting it for review or after getting it back with a shiny red rejection sticker. Preliminary review also helps us to identify potential areas of major weakness for an author so we can direct testers to test the right things or give an author advice/something to practice with.

 

d) Every Monday through Wednesday is panic mode central every week.

By having everything decided one way or another a week out, Monday through Wednesday should be smooth sailing. At the very least, the most recent test version will be available for recording in a pinch, since we wouldn’t clear the song for release if it was not.

 

As I mentioned in the intro blurb, all authors are stakeholders in the process, and I want to know how you would feel about the implementation of this process (things you like, things you dislike, potential problems I may have not considered, and so forth). I personally put a much higher value on quality over quantity for our releases, and while I understand that this might slow the release rate down a bit, I think we’re exceeding expectations and pushing out solid enough metrics as is… and if the charts are good enough, or get there through experience going through this process, we might not see much of a dip at all.

 

Please let me know your thoughts, and thank you for your consideration!

[TrojanNemo]

Even though I’m hardly an “author” anymore, I strongly agree with the proposed changes.

 

The internet seems to think C3 is the place to come for quality customs. Anything that helps us improve on quality is welcome.

[Gigakoops]

I feel that three weeks is ample time to get the charts done. And I agree that scheduling songs to test would be for the best, seeing how (at least while I’ve been here) I’ve seen several songs get little to no tests, and have to be fixed last minute due to some newly found major issue.

 

There’s one thing I am rather confused about, and that’s the preliminary tests. I understand what exactly they are and why we’re doing them, but I want to know how. Do we just post a thread for the song like normal, and just wait for an admin to do the preliminary tests before expecting other users to test it? Or do we send links to the files via PM to an admin? I’m not entirely sure on the process of submitting a song for preliminary testing vs. submitting one to normal playtesting.

[espher]

The process isn’t in place yet — this is just to to kind of discuss it — so I’m all for hearing what’s easiest for you guys.

 

I’m fine with people posting threads as normal, and then we reply with a note when a song is cleared (or lock it when it’s not, at which point the author can revisit and PM us when it’s updated). It should be pretty quick to bang through them as we don’t need to give detailed feedback or play parts over and over (that would be under playtesting), as we’re just trying to identify whether or not the song has significant or major issues at that point in the process.

 

I suppose another option would be creating a subforum where songs could be submitted and then moving them to playtesting proper when they’re cleared.

[ws54]

I pretty much agree with most of this. I certainly agree with all of the goals.

 

I’d like to suggest that authors should not even submit a song to play-testing until they consider it “complete,” with the exception of things that are optional for release such as a custom venue. In other words, the author should feel that the song is in a releaseable condition as is on the day it enters play-testing. That would mean all difficulties for all instruments are charted, practice sections added, all meta-data is correct, etc. I think this would encourage people to test the songs more and also earlier in the process. I understand that some authors might want opinions or help on a certain song before they reach that state, so another private forum topic could be added for that “Pre-release” purpose.

 

I am happy to help, but I don’t want people to come to rely on my help. Authors should not expect play-testers to do their rough passes for them. It is the authors job to do all of his work. For example, while occasional spelling, capitalization, and hyphenation problems can slip through, the song should not be filled with those types of errors. If it is then the author is not taking the time to do their own play-testing or is rushing through the development.

 

Finally, I’d like to see the secrecy of the administrator’s songs opened up for all in the private forums to play-test. Those songs should go through the same play-testing process as all of the others do. I see errors in those songs after release that could be caught with more eyeballs looking at them. The accuracy of the songs that I hope to play over and over again trumps the one-time Friday-surprise element hands down.

[Polish Weener]

all difficulties for all instruments are charted

 

I agree with the new playtesting process, but I do not agree with the above statement. When submitting a song, I believe the author should do all instruments on Expert (Pro Keys being an occasional exception) and basic things such as practice sections and Overdrive. The reason I believe an author should wait on reductions is to avoid making suggested changes to more than one difficulty. I suppose there’s no reason to say someone can’t submit reductions with the initial release, but I don’t think they should be required.

[ws54]

all difficulties for all instruments are charted

 

I agree with the new playtesting process, but I do not agree with the above statement. .

 

I find it to be trivial to change all difficulties at the same time since you are already right there making a change. Reductions are so easy to do. Having all difficulties will allow songs to be play-tested earlier and with more eyeballs since not everyone can play at an expert level. But whatever is decided upon is fine by me.

[Farottone]

Finally, I’d like to see the secrecy of the administrator’s songs opened up for all in the private forums to play-test. Those songs should go through the same play-testing process as all of the others do. I see errors in those songs after release that could be caught with more eyeballs looking at them. The accuracy of the songs that I hope to play over and over again trumps the one-time Friday-surprise element hands down.

 

It’s not “administrator’s songs”, it’s only my songs (most of them anyway): the fact that you can’t see our songs in the schedule doesn’t mean they’re not put through playtest.

[espher]

Would like some more feedback/thoughts on this, folks. The authors are the parties most impacted by the changes in the process and I’d rather know whether or not it appeals to you and/or you would be interested in making the commitments it would require, particularly from an authoring/scheduling/testing perspective, before any time is spent working towards its implementation.

 

The pitch as made in the OP is far from complete and omits details such as how playtesting would be assigned, what we would consider “acceptable” release criteria and “recommended” release criteria, and a number of other things we’ve been bouncing around over the past few weeks. This is just a piece in the puzzle, but I’m not going to mandate it without more input/feedback from the active authoring pool.

[EvilCornbread]

Seems totally reasonable to tighten up the process more. And it doesn’t seem like it’s much of an additional burden on authors, it’s just more clear what needs to happen when.

 

Might want to sort out how “packs” should work though — into playtesting individually and then earlier-finished songs held for the final song of the bunch, or playtested all at once?

[espher]

In the realm of the hypothetical, songs in packs should enter test as they are available to test, but the same sort of three-weeks-out-one-week-out check would be made for all components of the pack, and if one of the songs didn’t clear, the pack would be pulled/bumped, unless the author just wanted to go ahead with a couple of singles or the size would still be sufficient (e.g. splitting it into two 3-packs instead of one 6-pack).

[EvilCornbread]

A semi-related thing: it’s kind of a bummer that song threads get locked once they’ve been released. I get why using the bugtracker is nice, but there are, for example, a couple green pad toms on “Pride (In the name of love)” that are charted onto the left hand and I swear are basically impossible to hit on a pro setup. I’d be cool to have a more of a discussion about it with the input of multiple people rather than just entering a bug — especially since I don’t really know what the right solution is.

[pksage]

We (the C3 admins) have been talking about this subject a lot, and have come up with the following system that we’ll be using on a trial basis, pending your reactions and feedback. The following change does not address structured/assigned playtesting, and will coexist with that if we decide to do it.

 

Overview

 

To focus playtesting where it is most needed and expedite the process where it’s not, we’re revisiting an old C3 idea: “established authors”. Authors who regularly submit “good enough” v1s of their songs will be given a few special privileges. This doesn’t mean that the songs are perfect — just that they’re good enough for release, even if they need some more polish.

 

What special privileges? Well, the big change is that regular authors will no longer be able to schedule songs whenever they want. We’d rather schedule your songs only once rather than bump them, and we’ve had some last-minute discoveries of big bugs, which is bad for everyone. The general idea is that you’ll submit your song, and the community will use your comments to playtest instruments that you think need work. When you think it’s good enough for release (which might even be your v1), it will get a brief “sanity check” review. If it has big issues, we’ll point out the instruments that need work and put it back into playtesting. If it’s good enough for release, we’ll schedule it, ideally for the next week but at most 3 weeks away. Scheduled songs should always be considered release-worthy, and any playtesting should only reveal/fix minor issues that, if left as-is, would not compromise the song. As such, if we decide to do dedicated playtesting, we won’t assign people to scheduled songs. If you want to check out a scheduled song, go for it! You’re just doing it on your own time.

 

As you might have guessed, established authors will be able to schedule their songs wherever a hole exists in the calendar. The idea is that these authors will always generate “good enough” first versions, and they only need minor tweaks, if that. As an additional benefit, established authors may, at their discretion, skip public playtesting. If they choose to do this, we’re taking their word that they’re giving the songs a rigorous private review, ideally with at least one other person looking at it. If an established author releases several songs with medium- or large-sized bugs that have to be fixed post-release, they will most likely lose their “established author” status.

 

Our biggest concern with this system is that it will promote elitism and create a divide in the authoring community. That’s the last thing we want! Established authors will be heavily encouraged to support and nurture authors that are still learning, not sit in an ivory tower and sneer down at them. When an author releases several v1s that meet our standard of quality with no additional work required, they’ll be bumped up to “established author” status. Ideally, we want everyone to be an established author, because it means better charts and less testing all around, but no pressure! Your songs will always be released, no matter what. This is just a matter of how quickly they go through the system.

 

Our tentative list of established authors, in alphabetical order:

• espher
• farottone
• Nyxyxylyth
• Orange Harrison
• pksage
• RyanHYK
• Scienyde

…and we have many authors who are very close to the threshold. If you’re not on that list, please don’t take it personally! We know you guys work your asses off, and we’re hoping this change will be good for everyone. Ideally, this system will give us a better review process without adding too much structure or making customs feel like work.

 

“Wait, what? How exactly will submitting songs work now?”

 

Regular authors will use the following process, more or less:

 

1. You finish your song on all instruments, whether X-only or not.
2. You create a playtesting thread for it, with everything the same except for no release date.
3. You do one of the following:
   • If you think your v1 has no problem areas and is “good enough” for release, even if it’s not 100% perfect, say so in the OP of your playtesting thread or in a response to it.
   • Otherwise, note specific instruments that you think need playtesting.

[*]If you marked any instruments as needing playtests, we’ll try and get those looked at. We’re not sure if we’re doing a dedicated/assigned playtesting system yet, but something will be worked out.

[*]If you thought your song was ready to go or all marked instruments have been playtested, a C3 admin or skilled playtester will give your song a brief “sanity check” review for medium-to-large bugs.

• If your song is good enough to release as-is and includes reductions/etc., it will be scheduled to an official release date. Any playtesting after this point is optional, and exists for polishing purposes.
• If your song could release as-is but does not have reductions or is part of an incomplete pack, its release date will be marked as [TBD]. When you finish reductions or the rest of that pack’s songs are ready to go, we’ll move it to an official release date.
• If your song has significant issues, the reviewer will say which areas need work with a very brief summary of the problems (i.e. not a full, in-depth playtest). Your song will stay in playtest until those issues are addressed, at which point you’ll get another sanity check.

[*]Once scheduled, your song’s thread will stay in the playtesting forum until its release. People can continue to playtest it if they want, and if any major bugs are discovered that the sanity check missed, it will be pulled.

 

Established authors basically get to skip to step 6 (including being pulled if any major bugs are discovered).

 

“What does ‘good enough’ mean?”

 

We’re not demanding absolute perfection for C3 releases. Here’s some issues that would NOT stop a song from being “good enough”:

• One or two incorrect vocals pitches on short, talky-esque notes
• Infrequent lyric capitalization issues
• Vocal tubes starting/ending 1/64th or 2/64ths too early or too late
• Slightly misplaced vocal slides (+ note)
• One or two relative pitch issues during complex patterns in G/B/K
• Most sustain length problems
• One or two missing notes in G/B/K
• A few missing kick gems in Drums
• Not representing all notes in a chord in Pro Keys
• Pretty much any minor- or medium-sized issue on Easy, Medium, or Hard in any chart
• Most animation issues

But here’s some things that WOULD stop a song from being “good enough”:

• Lots of incorrect pitches, whether in lead vocals or a harmony part
• Repeated typos in lyrics
• Vocal tubes starting/ending 1/16th (4/64ths) or more too early or too late
• Prominent relative pitch issues in a G/B/K chart
• Many missing notes in G/B/K, or a note missing in a pattern repeated throughout the song
• Incorrect pitches in Pro Keys
• Blatant copy/paste errors in any chart
• Easy, Medium, or Hard being way too difficult (also includes chords in Easy, etc.)
• Missing [play] or [idle] events

Neither of those lists is all-inclusive, but you get the idea.

 

“What if I have songs scheduled right now?”

 

Any currently scheduled songs will stay scheduled and use the current system of “hope like hell that they’re good enough for release and bump them if we discover they’re not”.

 

“What about things that have to be scheduled in advance (holiday packs/etc.)?”

 

These will still be a thing. If you’re participating in a collaborative pack that will release on a specific date, just make sure your songs are ready ahead of time. All you need to do is pass the sanity check a week or so before that holiday.

 

This also applies to individual songs or your own packs that you want to release on a specific day. Even though you can’t schedule them directly, we’ll honor scheduling requests whenever possible. Your song(s) just need to pass that sanity check in time.

 

===========================================

 

These changes are going into effect over the next day or so, and we want to hear your feedback during this — did I mention? — TRIAL PERIOD. Please leave your comments in this thread, and continue to discuss the stuff addressed in the original post. Thanks!

[Gigakoops]

One thing I really dislike about the new Established Authors system is the option of letting songs skip public testing. It’s a bit counter-productive if you ask me. What may be “good enough” for the admins may not be “good enough” for a lot of people who play their given instrument well enough to notice the “one or two” inaccuracies. As someone who takes realistic charts seriously, it kinda scares me that letting charts skip public playtest is an option. I’d prefer we release things at as high a quality as possible instead of “good enough.”

 

We can still allow them to schedule their releases, etc., but please rethink this “skipping public playtesting” thing. And this is not meant to point out to any author at all, this is more of a word of caution.

[TrojanNemo]

This was discussed at length…

 

How is it any better if the song gets put on playtesting, and that instrument is not playtested? The song still goes out. You’re assuming that because it gets put in playtest, that every instrument gets playtested on every difficulty to the satisfaction of every potential player. That has not been supported by the past few months.

 

Established authors have proven their ability to chart songs accurately. If they happen to put out a song with a mistake, and someone catches it, then they can use the Bug Tracker and it will be fixed.

[Author]

Posts