Strange behavior on the Juniper MX240

Almost always direct upgrade works. If you ask TAC, they will likely
suggest a formal process and you'll be doing many upgrades, which
itself isn't actually something that is guaranteed to work (like in
WRL9 case, but that is vmhost RE, not yours).

And like Jordan said, you are out of resources but can extend them
with the command given, which should give you more run rate. You may
want to look in more detail how long you can keep running DPCE until
you're really out.

Ok thank you Harold we currently have the multihomed setup so we are still planning to address it in the bestway

For the Junos OS we can go 10.4 to 15.1 directly from the USB installer will be helpful also is their any tools available we can validate the config before upgrading it

Looks like you are out of FIB slots.

Would recommend reducing the number of routes you need to send into FIB, or upgrading to newer hardware that has more space.

Mark.

This feature was introduced in 10.4, so he should have it.

And yes, it's only supported for DPC's (I-chip).

Mark.

Curious, what RE are you running?

If you have DPC's still, I'd assume something like the RE-S-1300 or RE-S-2000, but not sure.

I ask because I'm not how late the older RE's can go.

Mark.

Certainly an option, but this requires quite a bit of babysitting, because as the DFZ oscillates, you can run into issues that send you into circles, largely unaware about FIB issues, especially when just a subset of routes are affected.

So yes, definitely an option, but the OP will need to watch the line cards like a hawk, and be ultra sensitive to debugging regular issues vs. FIB-related issues.

Mark.

Ok got it saku we will got with the direct upgrade of it

Ok Mark thank you for the suggestion we are currently running on the RE-S-1300 i am trying to check if juniper docs as the list of the DPCE with the I-Chip but not able to find it yet if you know DPCE model number let me know for it

If memory serves, I believe the DPC used I-chip for Layer 3, and EZ-chip for Layer 2 (Ethernet framing + MAC look-up). Someone with a longer memory may need to chime in. Mark.

ok mark got it currently we are using the following DPCE Cards

DPCE-R-4XGE-XFP

DPCE-R-Q-4XGE-XFP

When we tried using the command set chassis memory-enhanced route on the current Junos version 10.4 it says the command is not supported

so we will need to upgrade the Junos to the newer version in order to try above command of it.

Friend of mine had this issue recently on an MX chassis running DPC’s and RE-2000’s.

The extend memory command others have mentioned worked for him.

His instance drove us crazy for a bit. The device would learn a route, show that it was installed (show routes) but traffic to said prefix would bounce net unreachable. We even pushed a static just for S&G’s and that still didn’t resolve it. It was a single prefix that a customer had reported.

Some things to consider, as others have mentioned.

  1. IPv6 routes share the same space. And use more per-route. You can extend the life of this box (probably considerably) by dropping full tables for IPv6. Perhaps taking just a default (Same goes for v4).
  2. It seems from your previous output that you’re taking ~1 full v4 table. And 2x v6 tables. Do you really need a full table if you’re only taking 1 v4 table? Consider switching to a default only? In my Colleagues case, he was taking 2 full tables of v4 and v6 until he hit the same issue.
  3. While you’re RE’s could use a nice upgrade too. Your linecards are actually the problem here. If you move to anything > DPC you get the trio chipset with much more FIB space (2 Million routes I believe?). I’d consider new RE’s and new line cards for this box. Which might also mean new switch fabric controllers… Basically, we’d be talking a full overhaul sans the power supplies and chassis.
  4. Consider taking a default + full routes. Then filtering > /24 (if you even have anything < /24 learned now) (/48 on IPv6).
    Start with the memory command first and see where that gets you. But keep a watchful eye out for this to happen again (as the DFZ grows). Eventually your only option will be to filter routes and rely on a default or upgrade.

Hi Nick,

Thank you for the feedback on it. Would you please let me know which Juno OS version he had installed on the MX Chassis that works with the extended memory command of it?

Nehul,

He was running the 15 code train. I think 15.1R6.7. But don’t take that as fact. I just know it was 15 for sure.

ok got it thank you Nick

Odd, as this said it came alive in Junos 10.4:

Mark.

Yes Mark it is odd

These are the reasons why I was saying that while there may be some commands to move FIB allocations around, it’s a lot of admin. because the DFZ is very dynamic, and FIB programming issues due to lack of slots that affect different prefixes in different ways can have you chasing your tail for weeks before you figure things out. I think doing this should be a short-term solution as you make plans to get newer hardware. As a long-term strategy, it will tax day-to-day operations. Mark.

This seems like a strange position. The device has 16MB+16MB jtree
segments. The first is IP, the second is filters (Broadly).

OP has 16MB of first used.
OP has <5MB of second used.

What if the platform had originally shipped with a different balance
between filters and IP, and OP would have never hit this problem?

It is easy to see in many scenarios filter growth is negligible toi 0,
IP growth is not. OP would technically have 70% FIB growth left, so
DFZ of about 1.7M, which puts him in the year >2030 (potentially far
beyond, but at least that).

I view the recarving as fixing poorly dimensioned memory use. And had
it shipped with more sensible carving this discussion didn't exist,
and no one would suggest they are in any sort of tactical situation.
Saying there is a problem is logical fallacy, what if your platform
shipped carving of 1 prefix, and rest for filters, and you could do
50M+50M by config toggle. By your logic, this would be a tactical
temporary fix. No, we need to understand what we are doing, what is
the problem, what the solution is, we cannot categorically say this is
a tactical fix.

My response is to be taken in the context of running a (large) network, and not the view of a single box.

We have run into issues with platforms that have shipped with FIB's in favour of IPv4 and less for IPv6 and MPLS labels. Shifted around, you could give up whatever is left for IPv6 and ACL's to give more to IPv4, but you then end up losing native IPv6 scalability. And, of course, whatever other permutation you may think of that leaves you in a babysitting scenario for the protocol(s) assigned to peasantry.

When considered against the backdrop of a (large) network, one has to also consider the FIB requirements for the IGP, MPLS label space, e.t.c. And not to mention that IPv6 will require more FIB space than IPv4, both for the IGP and BGP.

I'd love to say people's ACL's are simple, but who knows what folk populate into every RADIUS PPPoE session that they think filters are a solution for?

So yes, it is important to understand the limitations (or capabilities) of your specific platform, but also look at the overall picture of your entire backbone, and get a full understanding of what re-juggling FIB memory may mean in the short and long term; of course, bearing in mind that for some operators, short-term could also be 10 years or more.

So all I'm saying is if there is a hack like this to help you delay moving to newer hardware, go for it. But know your hardware in the global context of your network, which will require a lot more attention to avoid getting caught out when you least expect it. I'd be remiss if I suggested that "implement, move on and forget" is a normal way to treat this hack.

Mark.

You are always here. You always need to understand your scale and how
much resources you have available, what is possible and what is not.
Existence or non-existence of configuration toggle to affect this
doesn't make that situation worse or better.
It is not implied to be a tactical short-term fix, it depends and you
must understand what you are doing to determine your position in any
case.