Optical Network Configurations

Thx. I wonder if switches like Diablo or TP-Link will accept mix and match SFP brands. I have 8 Finisars and they work in the FMCs so far.

The TP link definitely takes mix & match SFP ... I would expect the Diablo to also ... its possible you just got a bad SFP module?

The TP link definitely takes mix & match SFP ... I would expect the Diablo to also ... its possible you just got a bad SFP module?

As I mentioned earlier, I tried several (all Finisar) modules. And switched FMCs at the back end, etc. No errors, no nothing...just "disconnected". Tried setting static IP with subnet mask. Currently I have it disabled and using copper. It's is the HQ player server, not the NAA (which has FMC-to-FMC isolation) so it is theoretically less important, but the only pc I can install it on anyway. I'm gonna try Cisco SFPs and send signature8 a Finisar or two for him to try on his new TP-Link switch (to somewhat definitively answer the mix and match question)..

As I mentioned earlier, I tried several (all Finisar) modules. And switched FMCs at the back end, etc. No errors, no nothing...just "disconnected". Tried setting static IP with subnet mask. Currently I have it disabled and using copper. It's is the HQ player server, not the NAA (which has FMC-to-FMC isolation) so it is theoretically less important, but the only pc I can install it on anyway. I'm gonna try Cisco SFPs and send signature8 a Finisar or two for him to try on his new TP-Link switch (to somewhat definitively answer the mix and match question)..

I've used the TP-link & can confirm it works with Cisco / finisar etc SFPs.

You may have a problem with card or driver then. Does the card permit you to enter its driver config when it boots? If you can get the light on the TP link which corresponds to your connection then there is a network connection and the problem is a configuration issue ... otherwise it's possible the card is not working.

Driver config? The driver is one of the latest (4.8), which I downloaded (it auto found 4.1 and had same results, but documentation says 4.8 is minimally needed for WS2012). The pc sees the card and all it's driver info and settings. As stated earlier, the FMC sees the link, but it blinks a bit, not steady light (unlike my successful connections). But the FMC surely sees something, cuz when I disable or unplug the fiber the fx link light goes off. Currently the card has a static IP, but is moot cuz it has never been found anyway. DHCP was no better.

BTW, the card does not try to be the boot up device (unlike sig8's issues). BTW, don't worry about my copper ethernet connection called Jplay. That name is from a former setup. It's standard stuff. I have it plugged in so I can do remote desktop.

Ok so then the switch is seeing the card -- network cable unplugged would seem to be a configuration issue (the hardware doesn't think it's unplugged)

Driver config? The driver is one of the latest (4.8), which I downloaded (it auto found 4.1 and had same results, but documentation says 4.8 is minimally needed for WS2012). The pc sees the card and all it's driver info and settings. As stated earlier, the FMC sees the link, but it blinks a bit, not steady light (unlike my successful connections). But the FMC surely sees something, cuz when I disable or unplug the fiber the fx link light goes off. Currently the card has a static IP, but is moot cuz it has never been found anyway. DHCP was no better.

 

BTW, the card does not try to be the boot up device (unlike sig8's issues). BTW, don't worry about my copper ethernet connection called Jplay. That name is from a former setup. It's standard stuff. I have it plugged in so I can do remote desktop.

 

[ATTACH=CONFIG]19195[/ATTACH]

I am no networking expert, but I see your other card is part of Network4, has this card (IP) joined a network?

I can't find where to do that, sorry. I find no place where to add this card to network 4. This has become too big of a hijack to this educational thread.

I can't find where to do that, sorry. I find no place where to add this card to network 4. This has become too big of a hijack to this educational thread.

Windows can get spastic with 2 network cards -- lookup "windows multihomed network configuration"

Windows can get spastic with 2 network cards -- lookup "windows multihomed network configuration"

Yeah, much of my feedback was with the copper adapter disabled (dragged a monitor into the room). Still no luck. Maybe disabled isn't far enough for Windows.

Ted, are both Nics intended to be on the same subnet? If so, you can bridge them.

Ted, are both Nics intended to be on the same subnet? If so, you can bridge them.

That unfortunately gets me nowhere, cuz as soon as I disconnect the copper (my goal here, to eliminate copper noise and only use fiber) then the connection is lost anyway. Even when bridged the Mellanox shows as disconnected, no cable.

Got it. Have you ever successfully used the Mellanox? It would appear the problem is at the physical layer. The FMCs have a force/auto switch. Is there an equivalent here? Also are you sure the fiber and SFPs are all compatible in performance and mode?

Got it. Have you ever successfully used the Mellanox? It would appear the problem is at the physical layer. The FMCs have a force/auto switch. Is there an equivalent here? Also are you sure the fiber and SFPs are all compatible in performance and mode?

After you change the force or auto setting, make sure to power down and back up the units. They need to be set the same at both sides and are easily accidentally flipped. I ran into that problem before.

After you change the force or auto setting, make sure to power down and back up the units. They need to be set the same at both sides and are easily accidentally flipped. I ran into that problem before.

What do you mean "both sides" there is only one FMC in this example, the other end is the pc with a Mellanox card. And yes, I have tried both force and auto on the FMC, and unplug/plug the dc power each time I do that. Not sure where to tell Mellanox force or auto, frankly.

Lmitche, no the Mellanox was delivered a few days ago and I have not gotten it to work. We are suspecting a possible Finisar SFP incompatibility (suspecting it cuz I've run out of ideas) so I have some Ciscos on the way. The weirdest issue is that the FMC FX link sees it (but the light is not steady, it blinks steadily), the pc sees it and if I truly unplug the fiber the light goes out...but Windows says disconnected regardless..and there is no communication.

What do you mean "both sides" there is only one FMC in this example, the other end is the pc with a Mellanox card. And yes, I have tried both force and auto on the FMC, and unplug/plug the dc power each time I do that. Not sure where to tell Mellanox force or auto, frankly.

 

Lmitche, no the Mellanox was delivered a few days ago and I have not gotten it to work. We are suspecting a possible Finisar SFP incompatibility (suspecting it cuz I've run out of ideas) so I have some Ciscos on the way. The weirdest issue is that the FMC FX link sees it (but the light is not steady, it blinks steadily), the pc sees it and if I truly unplug the fiber the light goes out...but Windows says disconnected regardless..and there is no communication.

My apologies if you have seen this already, but this wikipedia page may be of help.

https://en.wikipedia.org/wiki/Gigabit_Ethernet

It shows the bewildering array of fiber standards. It was helpful to me in trying to understand compatibility issues at the physical level.

Yes, the assumption going in to the Mellanox (turned on to us by jabbr) is that it is indeed 100base-LX compatible, being that we already have SFPs into FMCs that are that spec. And the simple fact that Ashok (signature8) has the same card and it works. ? I will uncover whether the Finisar SFP is an issue when my Ciscos arrive tmrw.

Yes, the assumption going in to the Mellanox (turned on to us by jabbr) is that it is indeed 100base-LX compatible, being that we already have SFPs into FMCs that are that spec. And the simple fact that Ashok (signature8) has the same card and it works. ? I will uncover whether the Finisar SFP is an issue when my Ciscos arrive tmrw.

Well, I've never actually used Mellanox ethernet cards. I pay a bit more for Intel because Intel ethernet chips/cards/drivers have always worked better for me. Mellanox is a well respected company particularly in the Infiniband/HPC space but may require more expertise to get working (as evidenced here).

In my experience when the SFP is not compatible, the light doesn't go on. The fact that the light on the switch goes on suggests the physical link is up. My guess is that with the Cisco SFPs the same issue (lights on but no one home ). More likely to be a Windows configuration issue. I seem to have more trouble configuring Windows networks myself -- OS X just seems to work and Linux makes it easier to do manual config using a file.

Yes, the assumption going in to the Mellanox (turned on to us by jabbr) is that it is indeed 100base-LX compatible...

and I assume you mean 1000base-SX compatible, right? LX is the long range (singlemode and longer wavelength) variant (think km) whereas SX is the multimode 850nm variant (think 100m) ... a good and bad thing about SFP modules is that they give you lots and lots of options ...

Ted, the other issue might be firmware .. just thinking that your Mellanox might have Infiniband firmware and may need Ethernet firmware ... Mellanox started out with Infiniband and the concept of rDMA where a high performance cluster could effectively access memory on remote nodes ... the idea of memory to memory transfer over network. Still a niche product but generally high speed Ethernet has largest marketshare. Just looking at reasons why your particular card isn't working.

Jabbr, et al....

Today the Cisco SFPs arrived (a big $20 investment in four of them). I casually swapped 2 of them in for the Finisars (one in Mellanox card, one in FMC). Did nothing else. Expected little.......but...wait...what? Voila!!! We have connection, we have solid fx link light, we have success!!! Go freaking figure. And although there is no issue running both copper and fiber (multihomed) I pulled the copper one out, as per my goal. I have remote desktop running fine, etc. Yea!!!

So now I have both my HQP and my NAA on fiber. Next step is to demo signature8's switches (his quick first impression is the Diablo switch is quieter than the TP-Link one) and then get the NAS and my Signature Rendu on fiber, buying one of the same switches signature8 is using, and allowing me to repurpose two of my FMCs now being used at my copper switch end. In the meantime I will enjoy my even lower music noise floor. Thanks jabbr and others!

that's great news, Ted!

Jabbr, et al....

Today the Cisco SFPs arrived (a big $20 investment in four of them). I casually swapped 2 of them in for the Finisars (one in Mellanox card, one in FMC). Did nothing else. Expected little.......but...wait...what? Voila!!! We have connection, we have solid fx link light, we have success!!! Go freaking figure. And although there is no issue running both copper and fiber (multihomed) I pulled the copper one out, as per my goal. I have remote desktop running fine, etc. Yea!!!

 

So now I have both my HQP and my NAA on fiber. Next step is to demo signature8's switches (his quick first impression is the Diablo switch is quieter than the TP-Link one) and then get the NAS and my Signature Rendu on fiber, buying one of the same switches signature8 is using, and allowing me to repurpose two of my FMCs now being used at my copper switch end. In the meantime I will enjoy my even lower music noise floor. Thanks jabbr and others!

That's great. Can you post the exact model #s again of the SFPs that didn't work with the Mellanox and did? This is a learning experience for all of us. FWIW, I ordered a Mellanox card and ASRock Q1900M and I just received a Logic Supply MC600 case (with 2 pci-e expansion slots) for the NAA I'm building ... what a nice case!

The Mellanox card is an MNPA19-XTR ConnectX2 card

Mellanox Connectx 2 10GbE Ethernet Network Server Adapter MNPA19 XTR | eBay

The SFPs are:

A) Finisar 1000Base-SX FTRJ8519P1BNL-B1 Fiber Optic Transceiver Module This module worked in the FMC, (a TP Link MC220L) when the Cisco (see below) was in the Mellanox card. It did NOT work as the SFP for the Mellanox card, regardless of what SFP was in the FMC.

B) Cisco GLC-SX-MM 30-1301-02 Transceiver Module. This module worked in both the FMC and the Mellanox card.

So, in summary, the four combinations of Mellanox card to FMC:

a to a NO

a to b NO

b to a Yes

b to b Yes

Hi,

I follow your story carefully. Really interesting.

I just have 2 tplinks before my streamer, really good improvement. Based on that, I'm thinking about replacing the rather long RJ45 cables of my network with fibers.

Does someone experience a smoother/faster/more stable network with fibers ?

Thanks for this testing session !

I received both switches from Sig8 yesterday, the TP-Link TL-SG5412F and the Diablo 18198. I plugged four Finisar SFP's into the TP-Link, removed my copper switch, and plugged in the TP-Link. All went well, and now I have the HQP, NAA and my Synology NAS all connected via fiber. I will add the Rendu today. I will serious listen for one more night and then swap in the Diablo later tonight...and report back.