Displacement can be an excellent tool for creating surface detail, or to model things that would take far too long using regular methods. But in 3dsmax (and many other 3d apps), standard displacement has one big disadvantage: all it can do is take a point on a surface and push it outwards from the face normal. Displacement that doesn't follow the face normal is known as Non-Linear Displacement (or if the displacement still pushes the point straight out, but in a direction other than the face normal, this is called Vector Displacement).
For example, here's a displaced surface like you've probably seen before…
And here's something similar, except using non-linear displacement to warp the results in an axis that isn't perpendicular to the face normal.
Specialty Software for achieving Non Linear Displacement
Non-Linear Displacement has been around for quite some time, and there's tons of research on it, however, most 3d applications / renderers don't deal with it very well, if at all. This is partly because giving the user control over what direction the displacement goes is complex from a user interface perspective. Another reason is there doesn't seem to be enough interest from the average user to persuade the software companies to invest much money on the feature. If it's tough to do and few people would buy it, why bother doing it, right? So, non linear displacement has remained the domain of research and a few programmers willing to roll their own, but there are limited options for the standard artist.
In mentalray for max, you can use Vector Displacement maps, however, mentalray provides no method for creating the maps.
If you own the vray Renderer, you can produce Vector Displacement maps by taking the difference between a low res and highres mesh, baking it to a vector disp map, then using that map on your low res mesh to replciate the high res result. Go here for more info.
And here's an interesting thread on xsibase discussing how to use xsi and mentalray to do non-linear displacement.
Non Linear Displacement in 3dsmax with Scanline Renderer
This tutorial discusses a technique for doing non-linear displacement inside max without a special renderer.
Its basic principal is to displace your surface using the displace modifier (and not the displacement slot in the material) and then modify the direction of the resulting mesh using other modifiers. For this example, I'll displace a plane with a cellular texture…
But instead of just displacing out, I'll inflate the spheres so that the displacement rolls back overtop of itself, creating overlapping surfaces or under cuts…
Here's the max file to create this effect (max 8 format).
First, make a plane, apply a meshsmooth modifier with a lot of iterations, since we'll need lots of tiny faces to displace.
Next, apply your displace modifier, and place a cellular map in the displace slot. I adjusted the size of the cellular and few other parameters to get the following pattern.
Next you want to select the displaced cells. Use a volume select modifier. You must use volume select instead of say a poly select, because you want to select an area, not specific vertexes/faces. That way, if you need smother displacement, you can increase the number of meshsmooth iterations and your volume select modifier will still select the same stuff. For the volume select, I select the part of the plane that's not displaced, set it to vertex, then click invert so that I'm now selecting the vertex that make up the cells, then I apply a soft selection, and adjust the soft selection curve to my liking (experiment with this curve to get a variety of cool effects). The red and orange means those vertex are selected, the yellow and green are partly selected, and the blue are not selected at all.
Now apply a push modifier. Push will take a face and push it along its face normal. It will react with the soft selection by inflating the cells so instead of just bumps going straight up, you now have inflated nodules, like balloons coming out of your surface.
As a final touch, I put a polyselect modifier (to take myself out of subobject mode) followed by a relax modifier set to a few iterations on top in order to smooth out some of the displacement artifacts.
Here's another example file, squiggle.max (max 8 format), a squiggly spike. This uses the same principals, except I'm using an xform modifier instead of a push to modify the direction of my displacement.
Hopefully this will give you some of your own ideas on how to achieve non-linear displacements. Here's a few images where I've used the technique…