difference between -Xss and -XX:ThreadStackSize

There's basically no difference between -Xss and -XX:ThreadStackSize, except
that the former dates before HotSpot became the default JVM in Sun's JDK,
where as the latter is an internal VM flag of HotSpot. In effect, the former
is support in a lot of other JVMs, but the latter is specific to HotSpot.

As an historical aside, the command line arguments for Sun JDK 1.0.x and
1.1.x had these:
-ss<number>       set the C stack size of a process
-oss<number>      set the JAVA stack size of a process

These arguments got deprecated by -Xss and -Xoss in later versions of JDK.

Non-internal flags either get processd by the java launcher (such as
-server, -client, -verbosegc, etc.) or get converted into VM internal flags
by HotSpot. See hotspot/src/share/vm/runtime/arguments.cpp for details.

For example, see this version:
the launcher part: http://hg.openjdk.java.net/jdk7/jdk7/jdk/file/f097ca2434b1/src/share/bin/java.c lines 1052 - 1058,
and the VM part: http://hg.openjdk.java.net/jdk7/jdk7/hotspot/file/81d815b05abb/src/share/vm/runtime/arguments.cpp lines 2227 - 2240.

As you can see, -Xss gets converted into ThreadStackSize before the VM gets
to use it.
If the user did specify -ss or -Xss in command line arguments, that'll get
picked up directly by the launcher to run the main Java thread. Otherwise,
the launcher asks the VM for a preferred stack size, and HotSpot will return
ThreadStackSize, see here: http://hg.openjdk.java.net/jdk7/jdk7/hotspot/file/81d815b05abb/src/share/vm/prims/jni.cpp, lines 3286 - 3295.

That should explain how they've been implemented to be basically equivalent
in HotSpot.

One thing, though: because the -Xss/-XX:ThreadStackSize argument is handled
by the VM, *after* the primordial thread has been created, so they won't
cover the stack size of the primordial thread. In order to control stack
size correctly, Oracle/Sun's JDK doesn't run Java code in the primordial
thread. See this bug: http://bugs.sun.com/view_bug.do?bug_id=6316197. If you ever felt the function "ContinueInNewThread" in the launcher was
weird, that's the fix for this issue, so that Java code doesn't get run in
the primordial thread in HotSpot.

By the way,  I also want to know whether the java thread stack is equivalent
> to the native thread stack?  The stock HotSpot VM (the one in Oracle's Java SE JDK and OpenJDK) uses the
same stack for Java and native methods for a Java thread; Java frames and
native frames can be mixed together in such a stack.
-Xss/-XX:ThreadStackSize controls the whole stack's size for Java threads.
Because of this, -Xoss is simply ignored by HotSpot -- there's no separate
stack to set the size for.

There are some variants of HotSpot VM that does use a separate interpreter
stack, e.g. Azul's version. See this post from Cliff Click for more details
of the Azul implementation: http://www.azulsystems.com/blog/cliff/2010-04-06-jitd-code-calling-conventions-or-answering-plea-geekyness